{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/AI4Finance-LLC/FinRL-Library/blob/master/FinRL_single_stock_trading.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Idd1jem0TnST"
   },
   "source": [
    "# Deep Reinforcement Learning for Stock Trading from Scratch: Single Stock Trading\n",
    "\n",
    "Tutorials to use OpenAI DRL to trade single stock in one Jupyter Notebook | Presented at NeurIPS 2020: Deep RL Workshop\n",
    "\n",
    "* This blog is based on our paper: FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance, presented at NeurIPS 2020: Deep RL Workshop.\n",
    "* Check out medium blog for detailed explanations: https://towardsdatascience.com/finrl-for-quantitative-finance-tutorial-for-single-stock-trading-37d6d7c30aac\n",
    "* Please report any issues to our Github: https://github.com/AI4Finance-LLC/FinRL-Library/issues\n",
    "* **Pytorch Version** \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5vglc_9N5-KZ"
   },
   "source": [
    "## Content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ex2ord116AbP"
   },
   "source": [
    "* [1. Problem Definition](#0)\n",
    "* [2. Getting Started - Load Python packages](#1)\n",
    "    * [2.1. Install Packages](#1.1)    \n",
    "    * [2.2. Check Additional Packages](#1.2)\n",
    "    * [2.3. Import Packages](#1.3)\n",
    "    * [2.4. Create Folders](#1.4)\n",
    "* [3. Download Data](#2)\n",
    "* [4. Preprocess Data](#3)        \n",
    "    * [4.1. Technical Indicators](#3.1)\n",
    "    * [4.2. Perform Feature Engineering](#3.2)\n",
    "* [5.Build Environment](#4)  \n",
    "    * [5.1. Training & Trade Data Split](#4.1)\n",
    "    * [5.2. User-defined Environment](#4.2)   \n",
    "    * [5.3. Initialize Environment](#4.3)    \n",
    "* [6.Implement DRL Algorithms](#5)  \n",
    "* [7.Backtesting Performance](#6)  \n",
    "    * [7.1. BackTestStats](#6.1)\n",
    "    * [7.2. BackTestPlot](#6.2)   \n",
    "    * [7.3. Baseline Stats](#6.3)   \n",
    "    * [7.3. Compare to Stock Market Index](#6.4)             "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M1eLhMW36cLi"
   },
   "source": [
    "<a id='0'></a>\n",
    "# Part 1. Problem Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3eimeRv06YoK"
   },
   "source": [
    "This problem is to design an automated trading solution for single stock trading. We model the stock trading process as a Markov Decision Process (MDP). We then formulate our trading goal as a maximization problem.\n",
    "\n",
    "The components of the reinforcement learning environment are:\n",
    "\n",
    "\n",
    "* Action: The action space describes the allowed actions that the agent interacts with the\n",
    "environment. Normally, a ∈ A includes three actions: a ∈ {−1, 0, 1}, where −1, 0, 1 represent\n",
    "selling, holding, and buying one stock. Also, an action can be carried upon multiple shares. We use\n",
    "an action space {−k, ..., −1, 0, 1, ..., k}, where k denotes the number of shares. For example, \"Buy\n",
    "10 shares of AAPL\" or \"Sell 10 shares of AAPL\" are 10 or −10, respectively\n",
    "\n",
    "* Reward function: r(s, a, s′) is the incentive mechanism for an agent to learn a better action. The change of the portfolio value when action a is taken at state s and arriving at new state s',  i.e., r(s, a, s′) = v′ − v, where v′ and v represent the portfolio\n",
    "values at state s′ and s, respectively\n",
    "\n",
    "* State: The state space describes the observations that the agent receives from the environment. Just as a human trader needs to analyze various information before executing a trade, so\n",
    "our trading agent observes many different features to better learn in an interactive environment.\n",
    "\n",
    "* Environment: single stock trading for AAPL\n",
    "\n",
    "\n",
    "The data of the single stock that we will be using for this case study is obtained from Yahoo Finance API. The data contains Open-High-Low-Close price and volume.\n",
    "\n",
    "We use Apple Inc. stock: AAPL as an example throughout this article, because it is one of the most popular and profitable stocks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xD3f90UnTnSU"
   },
   "source": [
    "<a id='1'></a>\n",
    "# Part 2. Getting Started- Load Python Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UBUcBKap-oII"
   },
   "source": [
    "<a id='1.1'></a>\n",
    "## 2.1. Install all the packages through FinRL library\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "40axJBAP5mer",
    "outputId": "0e61993b-e962-4b70-a178-fe86404dd1ca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/AI4Finance-LLC/FinRL-Library.git\n",
      "  Cloning https://github.com/AI4Finance-LLC/FinRL-Library.git to c:\\users\\e0690420\\appdata\\local\\temp\\pip-req-build-lxk8tbrg\n",
      "Requirement already satisfied: numpy in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (1.19.2)\n",
      "Requirement already satisfied: pandas>=1.1.5 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (1.1.5)\n",
      "Requirement already satisfied: stockstats in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (0.3.2)\n",
      "Requirement already satisfied: yfinance in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (0.1.55)\n",
      "Requirement already satisfied: matplotlib in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (3.3.2)\n",
      "Requirement already satisfied: scikit-learn>=0.21.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (0.24.1)\n",
      "Requirement already satisfied: gym>=0.17 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (0.18.0)\n",
      "Requirement already satisfied: stable-baselines3[extra] in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (0.10.0)\n",
      "Requirement already satisfied: pytest in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (6.2.1)\n",
      "Requirement already satisfied: setuptools>=41.4.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (51.0.0.post20201207)\n",
      "Requirement already satisfied: wheel>=0.33.6 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from finrl==0.0.3) (0.36.2)\n",
      "Collecting pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2\n",
      "  Cloning https://github.com/quantopian/pyfolio.git to c:\\users\\e0690420\\appdata\\local\\temp\\pip-install-kzdlqp2z\\pyfolio_89bfdcc84ff445b494a905ea851f37ed\n",
      "Requirement already satisfied: ipython>=3.2.3 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (7.19.0)\n",
      "Requirement already satisfied: pytz>=2014.10 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (2020.4)\n",
      "Requirement already satisfied: scipy>=0.14.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (1.5.2)\n",
      "Requirement already satisfied: seaborn>=0.7.1 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (0.11.1)\n",
      "Requirement already satisfied: empyrical>=0.5.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (0.5.5)\n",
      "Requirement already satisfied: pandas-datareader>=0.2 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (0.9.0)\n",
      "Requirement already satisfied: traitlets>=4.2 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (5.0.5)\n",
      "Requirement already satisfied: pygments in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (2.7.3)\n",
      "Requirement already satisfied: decorator in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (4.4.2)\n",
      "Requirement already satisfied: jedi>=0.10 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (0.17.0)\n",
      "Requirement already satisfied: pickleshare in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (0.7.5)\n",
      "Requirement already satisfied: colorama in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (0.4.4)\n",
      "Requirement already satisfied: backcall in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (0.2.0)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (3.0.8)\n",
      "Requirement already satisfied: parso>=0.7.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from jedi>=0.10->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (0.8.1)\n",
      "Requirement already satisfied: certifi>=2020.06.20 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from matplotlib->finrl==0.0.3) (2020.12.5)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from matplotlib->finrl==0.0.3) (2.4.7)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from matplotlib->finrl==0.0.3) (2.8.1)\n",
      "Requirement already satisfied: pillow>=6.2.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from matplotlib->finrl==0.0.3) (8.0.1)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from matplotlib->finrl==0.0.3) (1.3.0)\n",
      "Requirement already satisfied: cycler>=0.10 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from matplotlib->finrl==0.0.3) (0.10.0)\n",
      "Requirement already satisfied: six in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from cycler>=0.10->matplotlib->finrl==0.0.3) (1.15.0)\n",
      "Requirement already satisfied: requests>=2.19.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (2.25.1)\n",
      "Requirement already satisfied: lxml in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (4.6.2)\n",
      "Requirement already satisfied: wcwidth in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (0.2.5)\n",
      "Requirement already satisfied: idna<3,>=2.5 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from requests>=2.19.0->pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (2.10)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from requests>=2.19.0->pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (4.0.0)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from requests>=2.19.0->pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (1.26.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from scikit-learn>=0.21.0->finrl==0.0.3) (2.1.0)\n",
      "Requirement already satisfied: joblib>=0.11 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from scikit-learn>=0.21.0->finrl==0.0.3) (1.0.0)\n",
      "Requirement already satisfied: ipython-genutils in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from traitlets>=4.2->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.0.3) (0.2.0)\n",
      "Requirement already satisfied: atomicwrites>=1.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pytest->finrl==0.0.3) (1.4.0)\n",
      "Requirement already satisfied: pluggy<1.0.0a1,>=0.12 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pytest->finrl==0.0.3) (0.13.1)\n",
      "Requirement already satisfied: iniconfig in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pytest->finrl==0.0.3) (1.1.1)\n",
      "Requirement already satisfied: py>=1.8.2 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pytest->finrl==0.0.3) (1.10.0)\n",
      "Requirement already satisfied: packaging in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pytest->finrl==0.0.3) (20.8)\n",
      "Requirement already satisfied: attrs>=19.2.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pytest->finrl==0.0.3) (20.3.0)\n",
      "Requirement already satisfied: importlib-metadata>=0.12 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pytest->finrl==0.0.3) (2.0.0)\n",
      "Requirement already satisfied: toml in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pytest->finrl==0.0.3) (0.10.1)\n",
      "Requirement already satisfied: zipp>=0.5 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from importlib-metadata>=0.12->pytest->finrl==0.0.3) (3.4.0)\n",
      "Requirement already satisfied: torch>=1.4.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from stable-baselines3[extra]->finrl==0.0.3) (1.7.1)\n",
      "Requirement already satisfied: cloudpickle in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from stable-baselines3[extra]->finrl==0.0.3) (1.6.0)\n",
      "Requirement already satisfied: psutil in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from stable-baselines3[extra]->finrl==0.0.3) (5.8.0)\n",
      "Requirement already satisfied: tensorboard in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from stable-baselines3[extra]->finrl==0.0.3) (2.4.1)\n",
      "Requirement already satisfied: atari-py~=0.2.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from stable-baselines3[extra]->finrl==0.0.3) (0.2.6)\n",
      "Requirement already satisfied: opencv-python in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from stable-baselines3[extra]->finrl==0.0.3) (4.5.1.48)\n",
      "Requirement already satisfied: typing-extensions in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from torch>=1.4.0->stable-baselines3[extra]->finrl==0.0.3) (3.7.4.3)\n",
      "Requirement already satisfied: int-date>=0.1.7 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from stockstats->finrl==0.0.3) (0.1.8)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.3 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from tensorboard->stable-baselines3[extra]->finrl==0.0.3) (1.24.0)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from tensorboard->stable-baselines3[extra]->finrl==0.0.3) (0.4.2)\n",
      "Requirement already satisfied: markdown>=2.6.8 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from tensorboard->stable-baselines3[extra]->finrl==0.0.3) (3.3.3)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from tensorboard->stable-baselines3[extra]->finrl==0.0.3) (1.8.0)\n",
      "Requirement already satisfied: absl-py>=0.4 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from tensorboard->stable-baselines3[extra]->finrl==0.0.3) (0.11.0)\n",
      "Requirement already satisfied: grpcio>=1.24.3 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from tensorboard->stable-baselines3[extra]->finrl==0.0.3) (1.35.0)\n",
      "Requirement already satisfied: protobuf>=3.6.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from tensorboard->stable-baselines3[extra]->finrl==0.0.3) (3.14.0)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from tensorboard->stable-baselines3[extra]->finrl==0.0.3) (1.0.1)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from google-auth<2,>=1.6.3->tensorboard->stable-baselines3[extra]->finrl==0.0.3) (4.7)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from google-auth<2,>=1.6.3->tensorboard->stable-baselines3[extra]->finrl==0.0.3) (4.2.1)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from google-auth<2,>=1.6.3->tensorboard->stable-baselines3[extra]->finrl==0.0.3) (0.2.8)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard->stable-baselines3[extra]->finrl==0.0.3) (1.3.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard->stable-baselines3[extra]->finrl==0.0.3) (0.4.8)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard->stable-baselines3[extra]->finrl==0.0.3) (3.1.0)\n",
      "Requirement already satisfied: multitasking>=0.0.7 in c:\\users\\e0690420\\anaconda3\\envs\\rl_fx\\lib\\site-packages (from yfinance->finrl==0.0.3) (0.0.9)\n"
     ]
    }
   ],
   "source": [
    "## install finrl library\n",
    "!pip install git+https://github.com/AI4Finance-LLC/FinRL-Library.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iXyHD6ir5sxk"
   },
   "source": [
    "\n",
    "<a id='1.2'></a>\n",
    "## 2.2. Check if the additional packages needed are present, if not install them. \n",
    "* Yahoo Finance API\n",
    "* pandas\n",
    "* numpy\n",
    "* matplotlib\n",
    "* stockstats\n",
    "* OpenAI gym\n",
    "* stable-baselines\n",
    "* tensorflow\n",
    "* pyfolio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QG17M4JwTnSZ"
   },
   "source": [
    "<a id='1.3'></a>\n",
    "## 2.3. Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0-0bsNMMTnSZ",
    "outputId": "e1c336ac-1280-4b16-b1d4-c2971d6667c2"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "matplotlib.use('Agg')\n",
    "import datetime\n",
    "\n",
    "from finrl.config import config\n",
    "from finrl.marketdata.yahoodownloader import YahooDownloader\n",
    "from finrl.preprocessing.preprocessors import FeatureEngineer\n",
    "from finrl.preprocessing.data import data_split\n",
    "from finrl.env.env_stocktrading import StockTradingEnv\n",
    "from finrl.model.models import DRLAgent\n",
    "from finrl.trade.backtest import backtest_stats, baseline_stats, backtest_plot\n",
    "\n",
    "import sys\n",
    "sys.path.append(\"../FinRL-Library\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "uIPbzYs1TnSd"
   },
   "outputs": [],
   "source": [
    "#Diable the warnings\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pos2IZAL54pp"
   },
   "source": [
    "<a id='1.4'></a>\n",
    "## 2.4. Create Folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "zp6lL6dZ53rX"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "if not os.path.exists(\"./\" + config.DATA_SAVE_DIR):\n",
    "    os.makedirs(\"./\" + config.DATA_SAVE_DIR)\n",
    "if not os.path.exists(\"./\" + config.TRAINED_MODEL_DIR):\n",
    "    os.makedirs(\"./\" + config.TRAINED_MODEL_DIR)\n",
    "if not os.path.exists(\"./\" + config.TENSORBOARD_LOG_DIR):\n",
    "    os.makedirs(\"./\" + config.TENSORBOARD_LOG_DIR)\n",
    "if not os.path.exists(\"./\" + config.RESULTS_DIR):\n",
    "    os.makedirs(\"./\" + config.RESULTS_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SBPM0sVvTnSg"
   },
   "source": [
    "<a id='2'></a>\n",
    "# Part 3. Download Data\n",
    "Yahoo Finance is a website that provides stock data, financial news, financial reports, etc. All the data provided by Yahoo Finance is free.\n",
    "* FinRL uses a class **YahooDownloader** to fetch data from Yahoo Finance API\n",
    "* Call Limit: Using the Public API (without authentication), you are limited to 2,000 requests per hour per IP (or up to a total of 48,000 requests a day).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GCGVmtGzjORf"
   },
   "source": [
    "\n",
    "\n",
    "-----\n",
    "class YahooDownloader:\n",
    "    Provides methods for retrieving daily stock data from\n",
    "    Yahoo Finance API\n",
    "\n",
    "    Attributes\n",
    "    ----------\n",
    "        start_date : str\n",
    "            start date of the data (modified from config.py)\n",
    "        end_date : str\n",
    "            end date of the data (modified from config.py)\n",
    "        ticker_list : list\n",
    "            a list of stock tickers (modified from config.py)\n",
    "\n",
    "    Methods\n",
    "    -------\n",
    "    fetch_data()\n",
    "        Fetches data from yahoo API\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "FBEiH5gOgMOx",
    "outputId": "de388576-0110-46f4-9257-dfe327b3eac5"
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'2000-01-01'"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "source": [
    "# from config.py start_date is a string\n",
    "config.START_DATE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "sWLMNQ8CgMRx",
    "outputId": "f1864c8b-2b5c-4867-9122-ccf83ebc2902"
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'2021-01-01'"
      ]
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "source": [
    "# from config.py end_date is a string\n",
    "config.END_DATE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AmFpuBEZhkF3"
   },
   "source": [
    "ticker_list is a list of stock tickers, in a single stock trading case, the list contains only 1 ticker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JtIFikNyTnSg",
    "outputId": "4c6a860f-6944-4728-984d-d0dc8d8f2c78"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[*********************100%***********************]  1 of 1 completed\n",
      "Shape of DataFrame:  (3022, 8)\n"
     ]
    }
   ],
   "source": [
    "# Download and save the data in a pandas DataFrame:\n",
    "data_df = YahooDownloader(start_date = '2009-01-01',\n",
    "                          end_date = '2021-01-01',\n",
    "                          ticker_list = ['AAPL']).fetch_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "E8ZKQmw6TnSl",
    "outputId": "954cbef4-f75b-4d4b-80f2-195eb13954ef"
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "(3022, 8)"
      ]
     },
     "metadata": {},
     "execution_count": 7
    }
   ],
   "source": [
    "data_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 195
    },
    "id": "Vi6D_ED6TnSs",
    "outputId": "2313cd51-596f-4a58-922b-5ec837bbcbda"
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "         date      open      high       low     close      volume   tic  day\n",
       "0  2008-12-31  3.070357  3.133571  3.047857  2.629544   607541200  AAPL    2\n",
       "1  2009-01-02  3.067143  3.251429  3.041429  2.795913   746015200  AAPL    4\n",
       "2  2009-01-05  3.327500  3.435000  3.311071  2.913912  1181608400  AAPL    0\n",
       "3  2009-01-06  3.426786  3.470357  3.299643  2.865849  1289310400  AAPL    1\n",
       "4  2009-01-07  3.278929  3.303571  3.223572  2.803923   753048800  AAPL    2"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>date</th>\n      <th>open</th>\n      <th>high</th>\n      <th>low</th>\n      <th>close</th>\n      <th>volume</th>\n      <th>tic</th>\n      <th>day</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2008-12-31</td>\n      <td>3.070357</td>\n      <td>3.133571</td>\n      <td>3.047857</td>\n      <td>2.629544</td>\n      <td>607541200</td>\n      <td>AAPL</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2009-01-02</td>\n      <td>3.067143</td>\n      <td>3.251429</td>\n      <td>3.041429</td>\n      <td>2.795913</td>\n      <td>746015200</td>\n      <td>AAPL</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2009-01-05</td>\n      <td>3.327500</td>\n      <td>3.435000</td>\n      <td>3.311071</td>\n      <td>2.913912</td>\n      <td>1181608400</td>\n      <td>AAPL</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2009-01-06</td>\n      <td>3.426786</td>\n      <td>3.470357</td>\n      <td>3.299643</td>\n      <td>2.865849</td>\n      <td>1289310400</td>\n      <td>AAPL</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2009-01-07</td>\n      <td>3.278929</td>\n      <td>3.303571</td>\n      <td>3.223572</td>\n      <td>2.803923</td>\n      <td>753048800</td>\n      <td>AAPL</td>\n      <td>2</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "source": [
    "data_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oWiqgpLzTnS3"
   },
   "source": [
    "<a id='3'></a>\n",
    "# Part 4. Preprocess Data\n",
    "Data preprocessing is a crucial step for training a high quality machine learning model. We need to check for missing data and do feature engineering in order to convert the data into a model-ready state.\n",
    "* FinRL uses a class **FeatureEngineer** to preprocess the data\n",
    "* Add **technical indicators**. In practical trading, various information needs to be taken into account, for example the historical stock prices, current holding shares, technical indicators, etc.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rJ9zmxpRks41"
   },
   "source": [
    "class FeatureEngineer:\n",
    "Provides methods for preprocessing the stock price data\n",
    "\n",
    "    Attributes\n",
    "    ----------\n",
    "        df: DataFrame\n",
    "            data downloaded from Yahoo API\n",
    "        feature_number : int\n",
    "            number of features we used\n",
    "        use_technical_indicator : boolean\n",
    "            we technical indicator or not\n",
    "        use_turbulence : boolean\n",
    "            use turbulence index or not\n",
    "\n",
    "    Methods\n",
    "    -------\n",
    "    preprocess_data()\n",
    "        main method to do the feature engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tHu7i-T_wRPc"
   },
   "source": [
    "<a id='3.1'></a>\n",
    "\n",
    "## 4.1 Technical Indicators\n",
    "* FinRL uses stockstats to calcualte technical indicators such as **Moving Average Convergence Divergence (MACD)**, **Relative Strength Index (RSI)**, **Average Directional Index (ADX)**, **Commodity Channel Index (CCI)** and other various indicators and stats.\n",
    "* **stockstats**: supplies a wrapper StockDataFrame based on the **pandas.DataFrame** with inline stock statistics/indicators support.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2RHwY1dHk09N",
    "outputId": "47be0df8-9e74-473e-f78e-9f267bcce3df"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['macd', 'boll_ub', 'boll_lb', 'rsi_30', 'cci_30', 'dx_30', 'close_30_sma', 'close_60_sma']\n"
     ]
    }
   ],
   "source": [
    "## we store the stockstats technical indicator column names in config.py\n",
    "tech_indicator_list=config.TECHNICAL_INDICATORS_LIST\n",
    "print(tech_indicator_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rImfGAfCkR8j",
    "outputId": "5b55c2c7-3217-49ad-95b7-d9af0ad91d7a"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['macd', 'boll_ub', 'boll_lb', 'rsi_30', 'cci_30', 'dx_30', 'close_30_sma', 'close_60_sma', 'kdjk', 'open_2_sma', 'boll', 'close_10.0_le_5_c', 'wr_10', 'dma', 'trix']\n"
     ]
    }
   ],
   "source": [
    "## user can add more technical indicators\n",
    "## check https://github.com/jealous/stockstats for different names\n",
    "tech_indicator_list=tech_indicator_list+['kdjk','open_2_sma','boll','close_10.0_le_5_c','wr_10','dma','trix']\n",
    "print(tech_indicator_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "etvRo2rSwZPg"
   },
   "source": [
    "<a id='3.2'></a>\n",
    "## 4.2 Perform Feature Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SAOsx0m-9u2k",
    "outputId": "6238a0f8-e0bb-4468-ec75-ab3381ea09a9"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Successfully added technical indicators\n"
     ]
    }
   ],
   "source": [
    "fe = FeatureEngineer(\n",
    "                    use_technical_indicator=True,\n",
    "                    tech_indicator_list = tech_indicator_list,\n",
    "                    use_turbulence=False,\n",
    "                    user_defined_feature = False)\n",
    "\n",
    "data_df = fe.preprocess_data(data_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 298
    },
    "id": "wytX_qwWMHP5",
    "outputId": "8113269b-39a0-462b-80b2-b8664eb27a66"
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "         date      open      high       low     close      volume   tic  day  \\\n",
       "0  2008-12-31  3.070357  3.133571  3.047857  2.629544   607541200  AAPL    2   \n",
       "1  2009-01-02  3.067143  3.251429  3.041429  2.795913   746015200  AAPL    4   \n",
       "2  2009-01-05  3.327500  3.435000  3.311071  2.913912  1181608400  AAPL    0   \n",
       "3  2009-01-06  3.426786  3.470357  3.299643  2.865849  1289310400  AAPL    1   \n",
       "4  2009-01-07  3.278929  3.303571  3.223572  2.803923   753048800  AAPL    2   \n",
       "\n",
       "       macd   boll_ub  ...       dx_30  close_30_sma  close_60_sma  \\\n",
       "0  0.000000  2.948009  ...  100.000000      2.629544      2.629544   \n",
       "1  0.003733  2.948009  ...  100.000000      2.712729      2.712729   \n",
       "2  0.008415  3.065525  ...  100.000000      2.779790      2.779790   \n",
       "3  0.008603  3.049973  ...  100.000000      2.801305      2.801305   \n",
       "4  0.006060  3.017194  ...   58.463049      2.801828      2.801828   \n",
       "\n",
       "         kdjk  open_2_sma      boll  close_10.0_le_5_c       wr_10  dma  \\\n",
       "0 -129.343666    3.070357  2.629544                1.0  588.030998  0.0   \n",
       "1 -125.199831    3.068750  2.712729                2.0  216.912162  0.0   \n",
       "2  -94.266522    3.197322  2.779790                3.0  132.399904  0.0   \n",
       "3  -76.489126    3.377143  2.801305                4.0  140.934334  0.0   \n",
       "4  -69.449994    3.352857  2.801828                5.0  155.371731  0.0   \n",
       "\n",
       "       trix  \n",
       "0  1.005511  \n",
       "1  1.005511  \n",
       "2  1.000508  \n",
       "3  0.836218  \n",
       "4  0.653105  \n",
       "\n",
       "[5 rows x 23 columns]"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>date</th>\n      <th>open</th>\n      <th>high</th>\n      <th>low</th>\n      <th>close</th>\n      <th>volume</th>\n      <th>tic</th>\n      <th>day</th>\n      <th>macd</th>\n      <th>boll_ub</th>\n      <th>...</th>\n      <th>dx_30</th>\n      <th>close_30_sma</th>\n      <th>close_60_sma</th>\n      <th>kdjk</th>\n      <th>open_2_sma</th>\n      <th>boll</th>\n      <th>close_10.0_le_5_c</th>\n      <th>wr_10</th>\n      <th>dma</th>\n      <th>trix</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2008-12-31</td>\n      <td>3.070357</td>\n      <td>3.133571</td>\n      <td>3.047857</td>\n      <td>2.629544</td>\n      <td>607541200</td>\n      <td>AAPL</td>\n      <td>2</td>\n      <td>0.000000</td>\n      <td>2.948009</td>\n      <td>...</td>\n      <td>100.000000</td>\n      <td>2.629544</td>\n      <td>2.629544</td>\n      <td>-129.343666</td>\n      <td>3.070357</td>\n      <td>2.629544</td>\n      <td>1.0</td>\n      <td>588.030998</td>\n      <td>0.0</td>\n      <td>1.005511</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2009-01-02</td>\n      <td>3.067143</td>\n      <td>3.251429</td>\n      <td>3.041429</td>\n      <td>2.795913</td>\n      <td>746015200</td>\n      <td>AAPL</td>\n      <td>4</td>\n      <td>0.003733</td>\n      <td>2.948009</td>\n      <td>...</td>\n      <td>100.000000</td>\n      <td>2.712729</td>\n      <td>2.712729</td>\n      <td>-125.199831</td>\n      <td>3.068750</td>\n      <td>2.712729</td>\n      <td>2.0</td>\n      <td>216.912162</td>\n      <td>0.0</td>\n      <td>1.005511</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2009-01-05</td>\n      <td>3.327500</td>\n      <td>3.435000</td>\n      <td>3.311071</td>\n      <td>2.913912</td>\n      <td>1181608400</td>\n      <td>AAPL</td>\n      <td>0</td>\n      <td>0.008415</td>\n      <td>3.065525</td>\n      <td>...</td>\n      <td>100.000000</td>\n      <td>2.779790</td>\n      <td>2.779790</td>\n      <td>-94.266522</td>\n      <td>3.197322</td>\n      <td>2.779790</td>\n      <td>3.0</td>\n      <td>132.399904</td>\n      <td>0.0</td>\n      <td>1.000508</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2009-01-06</td>\n      <td>3.426786</td>\n      <td>3.470357</td>\n      <td>3.299643</td>\n      <td>2.865849</td>\n      <td>1289310400</td>\n      <td>AAPL</td>\n      <td>1</td>\n      <td>0.008603</td>\n      <td>3.049973</td>\n      <td>...</td>\n      <td>100.000000</td>\n      <td>2.801305</td>\n      <td>2.801305</td>\n      <td>-76.489126</td>\n      <td>3.377143</td>\n      <td>2.801305</td>\n      <td>4.0</td>\n      <td>140.934334</td>\n      <td>0.0</td>\n      <td>0.836218</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2009-01-07</td>\n      <td>3.278929</td>\n      <td>3.303571</td>\n      <td>3.223572</td>\n      <td>2.803923</td>\n      <td>753048800</td>\n      <td>AAPL</td>\n      <td>2</td>\n      <td>0.006060</td>\n      <td>3.017194</td>\n      <td>...</td>\n      <td>58.463049</td>\n      <td>2.801828</td>\n      <td>2.801828</td>\n      <td>-69.449994</td>\n      <td>3.352857</td>\n      <td>2.801828</td>\n      <td>5.0</td>\n      <td>155.371731</td>\n      <td>0.0</td>\n      <td>0.653105</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 23 columns</p>\n</div>"
     },
     "metadata": {},
     "execution_count": 12
    }
   ],
   "source": [
    "data_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bwLhXo1cTnTQ"
   },
   "source": [
    "<a id='4'></a>\n",
    "# Part 5. Build Environment\n",
    "Considering the stochastic and interactive nature of the automated stock trading tasks, a financial task is modeled as a **Markov Decision Process (MDP)** problem. The training process involves observing stock price change, taking an action and reward's calculation to have the agent adjusting its strategy accordingly. By interacting with the environment, the trading agent will derive a trading strategy with the maximized rewards as time proceeds.\n",
    "\n",
    "Our trading environments, based on OpenAI Gym framework, simulate live stock markets with real market data according to the principle of time-driven simulation.\n",
    "\n",
    "The action space describes the allowed actions that the agent interacts with the environment. Normally, action a includes three actions: {-1, 0, 1}, where -1, 0, 1 represent selling, holding, and buying one share. Also, an action can be carried upon multiple shares. We use an action space {-k,…,-1, 0, 1, …, k}, where k denotes the number of shares to buy and -k denotes the number of shares to sell. For example, \"Buy 10 shares of AAPL\" or \"Sell 10 shares of AAPL\" are 10 or -10, respectively. The continuous action space needs to be normalized to [-1, 1], since the policy is defined on a Gaussian distribution, which needs to be normalized and symmetric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1D1FlBdOL4b3"
   },
   "source": [
    "<a id='4.1'></a>\n",
    "## 5.1 Training & Trade data split\n",
    "* Training: 2009-01-01 to 2018-12-31\n",
    "* Trade: 2019-01-01 to 2020-09-30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "xNOdqfTKL6K-"
   },
   "outputs": [],
   "source": [
    "#train = data_split(data_df, start = config.START_DATE, end = config.START_TRADE_DATE)\n",
    "#trade = data_split(data_df, start = config.START_TRADE_DATE, end = config.END_DATE)\n",
    "train = data_split(data_df, start = '2009-01-01', end = '2019-01-01')\n",
    "trade = data_split(data_df, start = '2019-01-01', end = '2021-01-01')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "KUxOshVouLXt"
   },
   "outputs": [],
   "source": [
    "## data normalization, this part is optional, have little impact\n",
    "#feaures_list = list(train.columns)\n",
    "#feaures_list.remove('date')\n",
    "#feaures_list.remove('tic')\n",
    "#feaures_list.remove('close')\n",
    "#print(feaures_list)\n",
    "#from sklearn import preprocessing\n",
    "#data_normaliser = preprocessing.StandardScaler()\n",
    "#train[feaures_list] = data_normaliser.fit_transform(train[feaures_list])\n",
    "#trade[feaures_list] = data_normaliser.transform(trade[feaures_list])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KMHyaSBBDGbe"
   },
   "source": [
    "<a id='4.2'></a>\n",
    "## 5.2 User-defined Environment: a simulation environment class "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "90V3S7cpDcQs"
   },
   "source": [
    "<a id='4.3'></a>\n",
    "## 5.3 Initialize Environment\n",
    "* **stock dimension**: the number of unique stock tickers we use\n",
    "* **hmax**: the maximum amount of shares to buy or sell\n",
    "* **initial amount**: the amount of money we use to trade in the begining\n",
    "* **transaction cost percentage**: a per share rate for every share trade\n",
    "* **tech_indicator_list**: a list of technical indicator names (modified from config.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OiH0xO96mGcL",
    "outputId": "490190d9-80dd-4ba0-c15f-2143cdd8341f"
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "['macd',\n",
       " 'boll_ub',\n",
       " 'boll_lb',\n",
       " 'rsi_30',\n",
       " 'cci_30',\n",
       " 'dx_30',\n",
       " 'close_30_sma',\n",
       " 'close_60_sma',\n",
       " 'kdjk',\n",
       " 'open_2_sma',\n",
       " 'boll',\n",
       " 'close_10.0_le_5_c',\n",
       " 'wr_10',\n",
       " 'dma',\n",
       " 'trix']"
      ]
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "source": [
    "## we store the stockstats technical indicator column names in config.py\n",
    "## check https://github.com/jealous/stockstats for different names\n",
    "tech_indicator_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GnWOKS7DyGM7",
    "outputId": "09dff84b-19e1-4c4a-c40f-d5280195eaf4"
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "metadata": {},
     "execution_count": 16
    }
   ],
   "source": [
    "# the stock dimension is 1, because we only use the price data of AAPL.\n",
    "len(train.tic.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ipJcnLvQGAba",
    "outputId": "dc32ebc6-2191-4541-99d2-7e9861da5df6"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Stock Dimension: 1, State Space: 11\n"
     ]
    }
   ],
   "source": [
    "stock_dimension = len(train.tic.unique())\n",
    "state_space = 1 + 2*stock_dimension + len(config.TECHNICAL_INDICATORS_LIST)*stock_dimension\n",
    "print(f\"Stock Dimension: {stock_dimension}, State Space: {state_space}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "JY5wTwYjGTrg"
   },
   "outputs": [],
   "source": [
    "env_kwargs = {\n",
    "    \"hmax\": 100, \n",
    "    \"initial_amount\": 100000, \n",
    "    \"buy_cost_pct\": 0.001, \n",
    "    \"sell_cost_pct\": 0.001, \n",
    "    \"state_space\": state_space, \n",
    "    \"stock_dim\": stock_dimension, \n",
    "    \"tech_indicator_list\": config.TECHNICAL_INDICATORS_LIST, \n",
    "    \"action_space\": stock_dimension, \n",
    "    \"reward_scaling\": 1e-4\n",
    "    \n",
    "}\n",
    "\n",
    "e_train_gym = StockTradingEnv(df = train, **env_kwargs)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vmyi2IPnyEkP",
    "outputId": "a968a96a-a1ee-426e-eed8-ee74d20b9bb8"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<class 'stable_baselines3.common.vec_env.dummy_vec_env.DummyVecEnv'>\n"
     ]
    }
   ],
   "source": [
    "env_train, _ = e_train_gym.get_sb_env()\n",
    "print(type(env_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mdnzYtM1TnTW"
   },
   "source": [
    "<a id='5'></a>\n",
    "# Part 6: Implement DRL Algorithms\n",
    "* The implementation of the DRL algorithms are based on **OpenAI Baselines** and **Stable Baselines**. Stable Baselines is a fork of OpenAI Baselines, with a major structural refactoring, and code cleanups.\n",
    "* FinRL library includes fine-tuned standard DRL algorithms, such as DQN, DDPG,\n",
    "Multi-Agent DDPG, PPO, SAC, A2C and TD3. We also allow users to\n",
    "design their own DRL algorithms by adapting these DRL algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "BRFIZDw8TnTX"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zD1NHzGyTnTc"
   },
   "source": [
    "### Model Training: 5 models, A2C DDPG, PPO, TD3, SAC\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y9CM5DeIr9GC"
   },
   "source": [
    "### Model 1: A2C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rOk-Mr73-EEq",
    "outputId": "1c56a5c6-91ad-4b81-dd07-24ae69e64c23"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0002}\nUsing cpu device\n"
     ]
    }
   ],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "\n",
    "A2C_PARAMS = {\"n_steps\": 5, \"ent_coef\": 0.005, \"learning_rate\": 0.0002}\n",
    "model_a2c = agent.get_model(model_name=\"a2c\",model_kwargs = A2C_PARAMS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DXHEidJh-E60",
    "outputId": "9ae1ede1-a8ee-4e56-c4f3-ef809e05d664"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Logging to tensorboard_log/a2c\\a2c_1\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 682      |\n",
      "|    iterations         | 100      |\n",
      "|    time_elapsed       | 0        |\n",
      "|    total_timesteps    | 500      |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -0.36    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 99       |\n",
      "|    policy_loss        | -0.193   |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.0121   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 682      |\n",
      "|    iterations         | 200      |\n",
      "|    time_elapsed       | 1        |\n",
      "|    total_timesteps    | 1000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -2.51    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 199      |\n",
      "|    policy_loss        | -1.51    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 1.47     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 682      |\n",
      "|    iterations         | 300      |\n",
      "|    time_elapsed       | 2        |\n",
      "|    total_timesteps    | 1500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -76.9    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 299      |\n",
      "|    policy_loss        | -1.06    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 1.34     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 682      |\n",
      "|    iterations         | 400      |\n",
      "|    time_elapsed       | 2        |\n",
      "|    total_timesteps    | 2000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -8.69    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 399      |\n",
      "|    policy_loss        | -0.906   |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.249    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 680      |\n",
      "|    iterations         | 500      |\n",
      "|    time_elapsed       | 3        |\n",
      "|    total_timesteps    | 2500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -95.1    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 499      |\n",
      "|    policy_loss        | 2.27     |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 2.79     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 5.75e+05 |\n",
      "|    total_cost         | 3.29e+03 |\n",
      "|    total_reward       | 4.75e+05 |\n",
      "|    total_reward_pct   | 475      |\n",
      "|    total_trades       | 2489     |\n",
      "| time/                 |          |\n",
      "|    fps                | 678      |\n",
      "|    iterations         | 600      |\n",
      "|    time_elapsed       | 4        |\n",
      "|    total_timesteps    | 3000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 599      |\n",
      "|    policy_loss        | 0.14     |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.0191   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 676      |\n",
      "|    iterations         | 700      |\n",
      "|    time_elapsed       | 5        |\n",
      "|    total_timesteps    | 3500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -52.5    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 699      |\n",
      "|    policy_loss        | 0.833    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.807    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 675      |\n",
      "|    iterations         | 800      |\n",
      "|    time_elapsed       | 5        |\n",
      "|    total_timesteps    | 4000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -18.5    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 799      |\n",
      "|    policy_loss        | 1.29     |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.8      |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 674      |\n",
      "|    iterations         | 900      |\n",
      "|    time_elapsed       | 6        |\n",
      "|    total_timesteps    | 4500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -4.07    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 899      |\n",
      "|    policy_loss        | -1.48    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.705    |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 672       |\n",
      "|    iterations         | 1000      |\n",
      "|    time_elapsed       | 7         |\n",
      "|    total_timesteps    | 5000      |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.43     |\n",
      "|    explained_variance | -1.56e+03 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 999       |\n",
      "|    policy_loss        | -2.71     |\n",
      "|    std                | 1.01      |\n",
      "|    value_loss         | 10.2      |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 4.93e+05 |\n",
      "|    total_cost         | 3.34e+03 |\n",
      "|    total_reward       | 3.93e+05 |\n",
      "|    total_reward_pct   | 393      |\n",
      "|    total_trades       | 2490     |\n",
      "| time/                 |          |\n",
      "|    fps                | 673      |\n",
      "|    iterations         | 1100     |\n",
      "|    time_elapsed       | 8        |\n",
      "|    total_timesteps    | 5500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -5.65    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 1099     |\n",
      "|    policy_loss        | 0.232    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.0503   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 673      |\n",
      "|    iterations         | 1200     |\n",
      "|    time_elapsed       | 8        |\n",
      "|    total_timesteps    | 6000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.44    |\n",
      "|    explained_variance | -791     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 1199     |\n",
      "|    policy_loss        | -0.958   |\n",
      "|    std                | 1.02     |\n",
      "|    value_loss         | 0.634    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 673      |\n",
      "|    iterations         | 1300     |\n",
      "|    time_elapsed       | 9        |\n",
      "|    total_timesteps    | 6500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.44    |\n",
      "|    explained_variance | -85.5    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 1299     |\n",
      "|    policy_loss        | 1.92     |\n",
      "|    std                | 1.02     |\n",
      "|    value_loss         | 2.17     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 673      |\n",
      "|    iterations         | 1400     |\n",
      "|    time_elapsed       | 10       |\n",
      "|    total_timesteps    | 7000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -101     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 1399     |\n",
      "|    policy_loss        | -0.336   |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.0298   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 673      |\n",
      "|    iterations         | 1500     |\n",
      "|    time_elapsed       | 11       |\n",
      "|    total_timesteps    | 7500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -128     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 1499     |\n",
      "|    policy_loss        | -1.07    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 4.15     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 5.87e+05 |\n",
      "|    total_cost         | 3.42e+03 |\n",
      "|    total_reward       | 4.87e+05 |\n",
      "|    total_reward_pct   | 487      |\n",
      "|    total_trades       | 2491     |\n",
      "| time/                 |          |\n",
      "|    fps                | 674      |\n",
      "|    iterations         | 1600     |\n",
      "|    time_elapsed       | 11       |\n",
      "|    total_timesteps    | 8000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -0.763   |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 1599     |\n",
      "|    policy_loss        | 0.285    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.0672   |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 673       |\n",
      "|    iterations         | 1700      |\n",
      "|    time_elapsed       | 12        |\n",
      "|    total_timesteps    | 8500      |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.43     |\n",
      "|    explained_variance | -3.39e+03 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 1699      |\n",
      "|    policy_loss        | -0.582    |\n",
      "|    std                | 1.01      |\n",
      "|    value_loss         | 0.216     |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 674      |\n",
      "|    iterations         | 1800     |\n",
      "|    time_elapsed       | 13       |\n",
      "|    total_timesteps    | 9000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -6.38    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 1799     |\n",
      "|    policy_loss        | 0.187    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.146    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 674      |\n",
      "|    iterations         | 1900     |\n",
      "|    time_elapsed       | 14       |\n",
      "|    total_timesteps    | 9500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -2.47    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 1899     |\n",
      "|    policy_loss        | -0.232   |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.0776   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 675      |\n",
      "|    iterations         | 2000     |\n",
      "|    time_elapsed       | 14       |\n",
      "|    total_timesteps    | 10000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -4.86    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 1999     |\n",
      "|    policy_loss        | -0.668   |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.963    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 4.98e+05 |\n",
      "|    total_cost         | 3.34e+03 |\n",
      "|    total_reward       | 3.98e+05 |\n",
      "|    total_reward_pct   | 398      |\n",
      "|    total_trades       | 2489     |\n",
      "| time/                 |          |\n",
      "|    fps                | 676      |\n",
      "|    iterations         | 2100     |\n",
      "|    time_elapsed       | 15       |\n",
      "|    total_timesteps    | 10500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -464     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 2099     |\n",
      "|    policy_loss        | 0.525    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.26     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 677      |\n",
      "|    iterations         | 2200     |\n",
      "|    time_elapsed       | 16       |\n",
      "|    total_timesteps    | 11000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -413     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 2199     |\n",
      "|    policy_loss        | -0.491   |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.189    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 678      |\n",
      "|    iterations         | 2300     |\n",
      "|    time_elapsed       | 16       |\n",
      "|    total_timesteps    | 11500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -4.31    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 2299     |\n",
      "|    policy_loss        | 0.874    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.513    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 678      |\n",
      "|    iterations         | 2400     |\n",
      "|    time_elapsed       | 17       |\n",
      "|    total_timesteps    | 12000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -4.07    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 2399     |\n",
      "|    policy_loss        | 0.466    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.0855   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 679      |\n",
      "|    iterations         | 2500     |\n",
      "|    time_elapsed       | 18       |\n",
      "|    total_timesteps    | 12500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -119     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 2499     |\n",
      "|    policy_loss        | 2.57     |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 5.12     |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| environment/          |           |\n",
      "|    portfolio_value    | 6.06e+05  |\n",
      "|    total_cost         | 3.4e+03   |\n",
      "|    total_reward       | 5.06e+05  |\n",
      "|    total_reward_pct   | 506       |\n",
      "|    total_trades       | 2494      |\n",
      "| time/                 |           |\n",
      "|    fps                | 680       |\n",
      "|    iterations         | 2600      |\n",
      "|    time_elapsed       | 19        |\n",
      "|    total_timesteps    | 13000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.43     |\n",
      "|    explained_variance | -4.37e+03 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 2599      |\n",
      "|    policy_loss        | -0.651    |\n",
      "|    std                | 1.01      |\n",
      "|    value_loss         | 0.189     |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 680      |\n",
      "|    iterations         | 2700     |\n",
      "|    time_elapsed       | 19       |\n",
      "|    total_timesteps    | 13500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -51.1    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 2699     |\n",
      "|    policy_loss        | 2.79     |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 2        |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 681      |\n",
      "|    iterations         | 2800     |\n",
      "|    time_elapsed       | 20       |\n",
      "|    total_timesteps    | 14000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -187     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 2799     |\n",
      "|    policy_loss        | 1.66     |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.556    |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 682       |\n",
      "|    iterations         | 2900      |\n",
      "|    time_elapsed       | 21        |\n",
      "|    total_timesteps    | 14500     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -1.36e+03 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 2899      |\n",
      "|    policy_loss        | -0.045    |\n",
      "|    std                | 1         |\n",
      "|    value_loss         | 0.132     |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 682      |\n",
      "|    iterations         | 3000     |\n",
      "|    time_elapsed       | 21       |\n",
      "|    total_timesteps    | 15000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -615     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 2999     |\n",
      "|    policy_loss        | 2.59     |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 9.79     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 6.48e+05 |\n",
      "|    total_cost         | 3.42e+03 |\n",
      "|    total_reward       | 5.48e+05 |\n",
      "|    total_reward_pct   | 548      |\n",
      "|    total_trades       | 2490     |\n",
      "| time/                 |          |\n",
      "|    fps                | 683      |\n",
      "|    iterations         | 3100     |\n",
      "|    time_elapsed       | 22       |\n",
      "|    total_timesteps    | 15500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 3099     |\n",
      "|    policy_loss        | 0.106    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.0113   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 683      |\n",
      "|    iterations         | 3200     |\n",
      "|    time_elapsed       | 23       |\n",
      "|    total_timesteps    | 16000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -193     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 3199     |\n",
      "|    policy_loss        | -0.672   |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.671    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 684      |\n",
      "|    iterations         | 3300     |\n",
      "|    time_elapsed       | 24       |\n",
      "|    total_timesteps    | 16500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.43    |\n",
      "|    explained_variance | -70.5    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 3299     |\n",
      "|    policy_loss        | 0.709    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.784    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 684      |\n",
      "|    iterations         | 3400     |\n",
      "|    time_elapsed       | 24       |\n",
      "|    total_timesteps    | 17000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -46.8    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 3399     |\n",
      "|    policy_loss        | 0.32     |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.286    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 684      |\n",
      "|    iterations         | 3500     |\n",
      "|    time_elapsed       | 25       |\n",
      "|    total_timesteps    | 17500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -742     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 3499     |\n",
      "|    policy_loss        | 0.566    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.371    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 6.59e+05 |\n",
      "|    total_cost         | 3.43e+03 |\n",
      "|    total_reward       | 5.59e+05 |\n",
      "|    total_reward_pct   | 559      |\n",
      "|    total_trades       | 2502     |\n",
      "| time/                 |          |\n",
      "|    fps                | 685      |\n",
      "|    iterations         | 3600     |\n",
      "|    time_elapsed       | 26       |\n",
      "|    total_timesteps    | 18000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 3599     |\n",
      "|    policy_loss        | -0.339   |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.155    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 685      |\n",
      "|    iterations         | 3700     |\n",
      "|    time_elapsed       | 26       |\n",
      "|    total_timesteps    | 18500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -243     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 3699     |\n",
      "|    policy_loss        | 1.65     |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 2.04     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 686      |\n",
      "|    iterations         | 3800     |\n",
      "|    time_elapsed       | 27       |\n",
      "|    total_timesteps    | 19000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 3799     |\n",
      "|    policy_loss        | 0.528    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.433    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 686      |\n",
      "|    iterations         | 3900     |\n",
      "|    time_elapsed       | 28       |\n",
      "|    total_timesteps    | 19500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -97.9    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 3899     |\n",
      "|    policy_loss        | -1.99    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 2.07     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 686      |\n",
      "|    iterations         | 4000     |\n",
      "|    time_elapsed       | 29       |\n",
      "|    total_timesteps    | 20000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -68.5    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 3999     |\n",
      "|    policy_loss        | -1.35    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 1.86     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 6.88e+05 |\n",
      "|    total_cost         | 3.47e+03 |\n",
      "|    total_reward       | 5.88e+05 |\n",
      "|    total_reward_pct   | 588      |\n",
      "|    total_trades       | 2497     |\n",
      "| time/                 |          |\n",
      "|    fps                | 686      |\n",
      "|    iterations         | 4100     |\n",
      "|    time_elapsed       | 29       |\n",
      "|    total_timesteps    | 20500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -24.2    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 4099     |\n",
      "|    policy_loss        | 0.763    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.495    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 687      |\n",
      "|    iterations         | 4200     |\n",
      "|    time_elapsed       | 30       |\n",
      "|    total_timesteps    | 21000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -409     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 4199     |\n",
      "|    policy_loss        | 0.257    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.114    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 687      |\n",
      "|    iterations         | 4300     |\n",
      "|    time_elapsed       | 31       |\n",
      "|    total_timesteps    | 21500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -56      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 4299     |\n",
      "|    policy_loss        | 0.803    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.589    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 687      |\n",
      "|    iterations         | 4400     |\n",
      "|    time_elapsed       | 31       |\n",
      "|    total_timesteps    | 22000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 4399     |\n",
      "|    policy_loss        | -0.62    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.643    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 687      |\n",
      "|    iterations         | 4500     |\n",
      "|    time_elapsed       | 32       |\n",
      "|    total_timesteps    | 22500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -9.58    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 4499     |\n",
      "|    policy_loss        | -0.312   |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.113    |\n",
      "------------------------------------\n",
      "day: 2515, episode: 10\n",
      "begin_total_asset: 100000.00\n",
      "end_total_asset: 665120.43\n",
      "total_reward: 565120.43\n",
      "total_cost: 3500.90\n",
      "total_trades: 2497\n",
      "Sharpe: 0.930\n",
      "=================================\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 6.65e+05 |\n",
      "|    total_cost         | 3.5e+03  |\n",
      "|    total_reward       | 5.65e+05 |\n",
      "|    total_reward_pct   | 565      |\n",
      "|    total_trades       | 2497     |\n",
      "| time/                 |          |\n",
      "|    fps                | 688      |\n",
      "|    iterations         | 4600     |\n",
      "|    time_elapsed       | 33       |\n",
      "|    total_timesteps    | 23000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -79.8    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 4599     |\n",
      "|    policy_loss        | -0.156   |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.0545   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 688      |\n",
      "|    iterations         | 4700     |\n",
      "|    time_elapsed       | 34       |\n",
      "|    total_timesteps    | 23500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -168     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 4699     |\n",
      "|    policy_loss        | -1.05    |\n",
      "|    std                | 0.998    |\n",
      "|    value_loss         | 1.63     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 688      |\n",
      "|    iterations         | 4800     |\n",
      "|    time_elapsed       | 34       |\n",
      "|    total_timesteps    | 24000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 4799     |\n",
      "|    policy_loss        | 0.269    |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 0.0735   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 688      |\n",
      "|    iterations         | 4900     |\n",
      "|    time_elapsed       | 35       |\n",
      "|    total_timesteps    | 24500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.41    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 4899     |\n",
      "|    policy_loss        | -0.396   |\n",
      "|    std                | 0.996    |\n",
      "|    value_loss         | 0.151    |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 689       |\n",
      "|    iterations         | 5000      |\n",
      "|    time_elapsed       | 36        |\n",
      "|    total_timesteps    | 25000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -4.77e+03 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 4999      |\n",
      "|    policy_loss        | 6.74      |\n",
      "|    std                | 0.997     |\n",
      "|    value_loss         | 25.2      |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 6.74e+05 |\n",
      "|    total_cost         | 3.42e+03 |\n",
      "|    total_reward       | 5.74e+05 |\n",
      "|    total_reward_pct   | 574      |\n",
      "|    total_trades       | 2496     |\n",
      "| time/                 |          |\n",
      "|    fps                | 688      |\n",
      "|    iterations         | 5100     |\n",
      "|    time_elapsed       | 37       |\n",
      "|    total_timesteps    | 25500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -644     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 5099     |\n",
      "|    policy_loss        | -0.733   |\n",
      "|    std                | 0.998    |\n",
      "|    value_loss         | 0.161    |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 689       |\n",
      "|    iterations         | 5200      |\n",
      "|    time_elapsed       | 37        |\n",
      "|    total_timesteps    | 26000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -3.66e+03 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 5199      |\n",
      "|    policy_loss        | 1.57      |\n",
      "|    std                | 0.998     |\n",
      "|    value_loss         | 2.62      |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 689      |\n",
      "|    iterations         | 5300     |\n",
      "|    time_elapsed       | 38       |\n",
      "|    total_timesteps    | 26500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -952     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 5299     |\n",
      "|    policy_loss        | 2.7      |\n",
      "|    std                | 0.999    |\n",
      "|    value_loss         | 4.55     |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 689       |\n",
      "|    iterations         | 5400      |\n",
      "|    time_elapsed       | 39        |\n",
      "|    total_timesteps    | 27000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -1.48e+04 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 5399      |\n",
      "|    policy_loss        | -1.5      |\n",
      "|    std                | 0.999     |\n",
      "|    value_loss         | 2.62      |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 689      |\n",
      "|    iterations         | 5500     |\n",
      "|    time_elapsed       | 39       |\n",
      "|    total_timesteps    | 27500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -1.1e+03 |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 5499     |\n",
      "|    policy_loss        | 1.32     |\n",
      "|    std                | 0.998    |\n",
      "|    value_loss         | 1.85     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 6.71e+05 |\n",
      "|    total_cost         | 3.46e+03 |\n",
      "|    total_reward       | 5.71e+05 |\n",
      "|    total_reward_pct   | 571      |\n",
      "|    total_trades       | 2495     |\n",
      "| time/                 |          |\n",
      "|    fps                | 689      |\n",
      "|    iterations         | 5600     |\n",
      "|    time_elapsed       | 40       |\n",
      "|    total_timesteps    | 28000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | 0.251    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 5599     |\n",
      "|    policy_loss        | -0.0089  |\n",
      "|    std                | 0.998    |\n",
      "|    value_loss         | 0.000229 |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 689       |\n",
      "|    iterations         | 5700      |\n",
      "|    time_elapsed       | 41        |\n",
      "|    total_timesteps    | 28500     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -1.17e+03 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 5699      |\n",
      "|    policy_loss        | 0.883     |\n",
      "|    std                | 0.996     |\n",
      "|    value_loss         | 0.699     |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 689      |\n",
      "|    iterations         | 5800     |\n",
      "|    time_elapsed       | 42       |\n",
      "|    total_timesteps    | 29000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -140     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 5799     |\n",
      "|    policy_loss        | 0.0623   |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 0.0453   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 689      |\n",
      "|    iterations         | 5900     |\n",
      "|    time_elapsed       | 42       |\n",
      "|    total_timesteps    | 29500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -159     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 5899     |\n",
      "|    policy_loss        | -0.773   |\n",
      "|    std                | 0.998    |\n",
      "|    value_loss         | 0.187    |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 689       |\n",
      "|    iterations         | 6000      |\n",
      "|    time_elapsed       | 43        |\n",
      "|    total_timesteps    | 30000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -3.96e+05 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 5999      |\n",
      "|    policy_loss        | -3.45     |\n",
      "|    std                | 0.997     |\n",
      "|    value_loss         | 6.01      |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 6.45e+05 |\n",
      "|    total_cost         | 3.5e+03  |\n",
      "|    total_reward       | 5.45e+05 |\n",
      "|    total_reward_pct   | 545      |\n",
      "|    total_trades       | 2492     |\n",
      "| time/                 |          |\n",
      "|    fps                | 689      |\n",
      "|    iterations         | 6100     |\n",
      "|    time_elapsed       | 44       |\n",
      "|    total_timesteps    | 30500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 6099     |\n",
      "|    policy_loss        | -0.129   |\n",
      "|    std                | 0.998    |\n",
      "|    value_loss         | 0.0254   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 690      |\n",
      "|    iterations         | 6200     |\n",
      "|    time_elapsed       | 44       |\n",
      "|    total_timesteps    | 31000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.41    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 6199     |\n",
      "|    policy_loss        | 0.759    |\n",
      "|    std                | 0.995    |\n",
      "|    value_loss         | 0.64     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 690      |\n",
      "|    iterations         | 6300     |\n",
      "|    time_elapsed       | 45       |\n",
      "|    total_timesteps    | 31500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.41    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 6299     |\n",
      "|    policy_loss        | -0.139   |\n",
      "|    std                | 0.996    |\n",
      "|    value_loss         | 0.0127   |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 690       |\n",
      "|    iterations         | 6400      |\n",
      "|    time_elapsed       | 46        |\n",
      "|    total_timesteps    | 32000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -3.17e+03 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 6399      |\n",
      "|    policy_loss        | 1.69      |\n",
      "|    std                | 0.997     |\n",
      "|    value_loss         | 1.98      |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 690      |\n",
      "|    iterations         | 6500     |\n",
      "|    time_elapsed       | 47       |\n",
      "|    total_timesteps    | 32500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 6499     |\n",
      "|    policy_loss        | 2.17     |\n",
      "|    std                | 0.998    |\n",
      "|    value_loss         | 4.44     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 6.92e+05 |\n",
      "|    total_cost         | 3.52e+03 |\n",
      "|    total_reward       | 5.92e+05 |\n",
      "|    total_reward_pct   | 592      |\n",
      "|    total_trades       | 2505     |\n",
      "| time/                 |          |\n",
      "|    fps                | 690      |\n",
      "|    iterations         | 6600     |\n",
      "|    time_elapsed       | 47       |\n",
      "|    total_timesteps    | 33000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 6599     |\n",
      "|    policy_loss        | -0.13    |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 0.0353   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 690      |\n",
      "|    iterations         | 6700     |\n",
      "|    time_elapsed       | 48       |\n",
      "|    total_timesteps    | 33500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 6699     |\n",
      "|    policy_loss        | 0.0619   |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 0.102    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 690      |\n",
      "|    iterations         | 6800     |\n",
      "|    time_elapsed       | 49       |\n",
      "|    total_timesteps    | 34000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.41    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 6799     |\n",
      "|    policy_loss        | 1.43     |\n",
      "|    std                | 0.995    |\n",
      "|    value_loss         | 1.02     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 690      |\n",
      "|    iterations         | 6900     |\n",
      "|    time_elapsed       | 49       |\n",
      "|    total_timesteps    | 34500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 6899     |\n",
      "|    policy_loss        | -0.173   |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 0.199    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 7000     |\n",
      "|    time_elapsed       | 50       |\n",
      "|    total_timesteps    | 35000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 6999     |\n",
      "|    policy_loss        | -5.27    |\n",
      "|    std                | 0.999    |\n",
      "|    value_loss         | 11.9     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 6.99e+05 |\n",
      "|    total_cost         | 3.62e+03 |\n",
      "|    total_reward       | 5.99e+05 |\n",
      "|    total_reward_pct   | 599      |\n",
      "|    total_trades       | 2500     |\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 7100     |\n",
      "|    time_elapsed       | 51       |\n",
      "|    total_timesteps    | 35500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 7099     |\n",
      "|    policy_loss        | -0.645   |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 0.35     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 7200     |\n",
      "|    time_elapsed       | 52       |\n",
      "|    total_timesteps    | 36000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -208     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 7199     |\n",
      "|    policy_loss        | 1.31     |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 1.22     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 7300     |\n",
      "|    time_elapsed       | 52       |\n",
      "|    total_timesteps    | 36500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 7299     |\n",
      "|    policy_loss        | 0.129    |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 0.125    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 7400     |\n",
      "|    time_elapsed       | 53       |\n",
      "|    total_timesteps    | 37000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -276     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 7399     |\n",
      "|    policy_loss        | -0.999   |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 0.694    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 7500     |\n",
      "|    time_elapsed       | 54       |\n",
      "|    total_timesteps    | 37500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -684     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 7499     |\n",
      "|    policy_loss        | 0.21     |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 0.178    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 7.11e+05 |\n",
      "|    total_cost         | 3.47e+03 |\n",
      "|    total_reward       | 6.11e+05 |\n",
      "|    total_reward_pct   | 611      |\n",
      "|    total_trades       | 2496     |\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 7600     |\n",
      "|    time_elapsed       | 54       |\n",
      "|    total_timesteps    | 38000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -455     |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 7599     |\n",
      "|    policy_loss        | -0.401   |\n",
      "|    std                | 0.997    |\n",
      "|    value_loss         | 0.0532   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 7700     |\n",
      "|    time_elapsed       | 55       |\n",
      "|    total_timesteps    | 38500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 7699     |\n",
      "|    policy_loss        | -0.0306  |\n",
      "|    std                | 0.996    |\n",
      "|    value_loss         | 0.0167   |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 691       |\n",
      "|    iterations         | 7800      |\n",
      "|    time_elapsed       | 56        |\n",
      "|    total_timesteps    | 39000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -8.76e+06 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 7799      |\n",
      "|    policy_loss        | -0.0635   |\n",
      "|    std                | 1         |\n",
      "|    value_loss         | 0.334     |\n",
      "-------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 691       |\n",
      "|    iterations         | 7900      |\n",
      "|    time_elapsed       | 57        |\n",
      "|    total_timesteps    | 39500     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -1.38e+07 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 7899      |\n",
      "|    policy_loss        | -1.1      |\n",
      "|    std                | 1         |\n",
      "|    value_loss         | 1.41      |\n",
      "-------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 691       |\n",
      "|    iterations         | 8000      |\n",
      "|    time_elapsed       | 57        |\n",
      "|    total_timesteps    | 40000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -4.19e+12 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 7999      |\n",
      "|    policy_loss        | 1.01      |\n",
      "|    std                | 1         |\n",
      "|    value_loss         | 1.05      |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 7.29e+05 |\n",
      "|    total_cost         | 3.56e+03 |\n",
      "|    total_reward       | 6.29e+05 |\n",
      "|    total_reward_pct   | 629      |\n",
      "|    total_trades       | 2498     |\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 8100     |\n",
      "|    time_elapsed       | 58       |\n",
      "|    total_timesteps    | 40500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -27.9    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 8099     |\n",
      "|    policy_loss        | 0.134    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.0286   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 8200     |\n",
      "|    time_elapsed       | 59       |\n",
      "|    total_timesteps    | 41000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 8199     |\n",
      "|    policy_loss        | -0.1     |\n",
      "|    std                | 0.999    |\n",
      "|    value_loss         | 0.0329   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 691      |\n",
      "|    iterations         | 8300     |\n",
      "|    time_elapsed       | 59       |\n",
      "|    total_timesteps    | 41500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 8299     |\n",
      "|    policy_loss        | 0.745    |\n",
      "|    std                | 0.999    |\n",
      "|    value_loss         | 0.799    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 8400     |\n",
      "|    time_elapsed       | 60       |\n",
      "|    total_timesteps    | 42000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 8399     |\n",
      "|    policy_loss        | -0.392   |\n",
      "|    std                | 0.999    |\n",
      "|    value_loss         | 0.0985   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 8500     |\n",
      "|    time_elapsed       | 61       |\n",
      "|    total_timesteps    | 42500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 8499     |\n",
      "|    policy_loss        | 1.58     |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 3.04     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 6.9e+05  |\n",
      "|    total_cost         | 3.5e+03  |\n",
      "|    total_reward       | 5.9e+05  |\n",
      "|    total_reward_pct   | 590      |\n",
      "|    total_trades       | 2495     |\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 8600     |\n",
      "|    time_elapsed       | 62       |\n",
      "|    total_timesteps    | 43000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -1.56    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 8599     |\n",
      "|    policy_loss        | -0.596   |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.269    |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 8700     |\n",
      "|    time_elapsed       | 62       |\n",
      "|    total_timesteps    | 43500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -4.6e+12 |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 8699     |\n",
      "|    policy_loss        | -0.883   |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 1.16     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 8800     |\n",
      "|    time_elapsed       | 63       |\n",
      "|    total_timesteps    | 44000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -13.3    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 8799     |\n",
      "|    policy_loss        | -0.0709  |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.0759   |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 8900     |\n",
      "|    time_elapsed       | 64       |\n",
      "|    total_timesteps    | 44500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -96.3    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 8899     |\n",
      "|    policy_loss        | 0.345    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.18     |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 692       |\n",
      "|    iterations         | 9000      |\n",
      "|    time_elapsed       | 65        |\n",
      "|    total_timesteps    | 45000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -1.77e+05 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 8999      |\n",
      "|    policy_loss        | 3.71      |\n",
      "|    std                | 1         |\n",
      "|    value_loss         | 17        |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 7.21e+05 |\n",
      "|    total_cost         | 3.48e+03 |\n",
      "|    total_reward       | 6.21e+05 |\n",
      "|    total_reward_pct   | 621      |\n",
      "|    total_trades       | 2493     |\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 9100     |\n",
      "|    time_elapsed       | 65       |\n",
      "|    total_timesteps    | 45500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -3.24    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 9099     |\n",
      "|    policy_loss        | -0.632   |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.343    |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 692       |\n",
      "|    iterations         | 9200      |\n",
      "|    time_elapsed       | 66        |\n",
      "|    total_timesteps    | 46000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -8.89e+04 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 9199      |\n",
      "|    policy_loss        | -2.91     |\n",
      "|    std                | 1         |\n",
      "|    value_loss         | 1.87      |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 9300     |\n",
      "|    time_elapsed       | 67       |\n",
      "|    total_timesteps    | 46500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -49.6    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 9299     |\n",
      "|    policy_loss        | 0.916    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.28     |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 692       |\n",
      "|    iterations         | 9400      |\n",
      "|    time_elapsed       | 67        |\n",
      "|    total_timesteps    | 47000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -1.02e+04 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 9399      |\n",
      "|    policy_loss        | -0.764    |\n",
      "|    std                | 1         |\n",
      "|    value_loss         | 0.379     |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 9500     |\n",
      "|    time_elapsed       | 68       |\n",
      "|    total_timesteps    | 47500    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | -44.7    |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 9499     |\n",
      "|    policy_loss        | 0.632    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.388    |\n",
      "------------------------------------\n",
      "day: 2515, episode: 20\n",
      "begin_total_asset: 100000.00\n",
      "end_total_asset: 713392.85\n",
      "total_reward: 613392.85\n",
      "total_cost: 3600.38\n",
      "total_trades: 2500\n",
      "Sharpe: 0.951\n",
      "=================================\n",
      "------------------------------------\n",
      "| environment/          |          |\n",
      "|    portfolio_value    | 7.13e+05 |\n",
      "|    total_cost         | 3.6e+03  |\n",
      "|    total_reward       | 6.13e+05 |\n",
      "|    total_reward_pct   | 613      |\n",
      "|    total_trades       | 2500     |\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 9600     |\n",
      "|    time_elapsed       | 69       |\n",
      "|    total_timesteps    | 48000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 9599     |\n",
      "|    policy_loss        | 0.203    |\n",
      "|    std                | 1        |\n",
      "|    value_loss         | 0.0287   |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 692       |\n",
      "|    iterations         | 9700      |\n",
      "|    time_elapsed       | 70        |\n",
      "|    total_timesteps    | 48500     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -6.19e+11 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 9699      |\n",
      "|    policy_loss        | -1.05     |\n",
      "|    std                | 1         |\n",
      "|    value_loss         | 0.53      |\n",
      "-------------------------------------\n",
      "------------------------------------\n",
      "| time/                 |          |\n",
      "|    fps                | 692      |\n",
      "|    iterations         | 9800     |\n",
      "|    time_elapsed       | 70       |\n",
      "|    total_timesteps    | 49000    |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -1.42    |\n",
      "|    explained_variance | nan      |\n",
      "|    learning_rate      | 0.0002   |\n",
      "|    n_updates          | 9799     |\n",
      "|    policy_loss        | 0.419    |\n",
      "|    std                | 1.01     |\n",
      "|    value_loss         | 0.325    |\n",
      "------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 692       |\n",
      "|    iterations         | 9900      |\n",
      "|    time_elapsed       | 71        |\n",
      "|    total_timesteps    | 49500     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -6.42e+03 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 9899      |\n",
      "|    policy_loss        | -0.795    |\n",
      "|    std                | 1.01      |\n",
      "|    value_loss         | 0.931     |\n",
      "-------------------------------------\n",
      "-------------------------------------\n",
      "| time/                 |           |\n",
      "|    fps                | 692       |\n",
      "|    iterations         | 10000     |\n",
      "|    time_elapsed       | 72        |\n",
      "|    total_timesteps    | 50000     |\n",
      "| train/                |           |\n",
      "|    entropy_loss       | -1.42     |\n",
      "|    explained_variance | -1.28e+03 |\n",
      "|    learning_rate      | 0.0002    |\n",
      "|    n_updates          | 9999      |\n",
      "|    policy_loss        | 0.0125    |\n",
      "|    std                | 1.01      |\n",
      "|    value_loss         | 0.337     |\n",
      "-------------------------------------\n"
     ]
    }
   ],
   "source": [
    "trained_a2c = agent.train_model(model=model_a2c, \n",
    "                                tb_log_name='a2c',\n",
    "                                total_timesteps=50000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9upN8FI2r_X1"
   },
   "source": [
    "### Model 2: DDPG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gUFJlHsi-Ka-",
    "outputId": "1b7db70f-f740-401c-ab55-a567b4235e18"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'batch_size': 64, 'buffer_size': 500000, 'learning_rate': 0.0001}\nUsing cpu device\n"
     ]
    }
   ],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "DDPG_PARAMS = {\"batch_size\": 64, \"buffer_size\": 500000, \"learning_rate\": 0.0001}\n",
    "\n",
    "\n",
    "model_ddpg = agent.get_model(\"ddpg\",model_kwargs = DDPG_PARAMS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "I5J_Scwp-Nis",
    "outputId": "a538962d-33c8-44d3-94af-6ef61fc0d230"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Logging to tensorboard_log/ddpg\\ddpg_1\n",
      "----------------------------------\n",
      "| environment/        |          |\n",
      "|    portfolio_value  | 1e+05    |\n",
      "|    total_cost       | 0        |\n",
      "|    total_reward     | 0        |\n",
      "|    total_reward_pct | 0        |\n",
      "|    total_trades     | 0        |\n",
      "| time/               |          |\n",
      "|    episodes         | 4        |\n",
      "|    fps              | 155      |\n",
      "|    time_elapsed     | 64       |\n",
      "|    total timesteps  | 10064    |\n",
      "| train/              |          |\n",
      "|    actor_loss       | -167     |\n",
      "|    critic_loss      | 1.03e+03 |\n",
      "|    learning_rate    | 0.0001   |\n",
      "|    n_updates        | 7548     |\n",
      "----------------------------------\n",
      "----------------------------------\n",
      "| environment/        |          |\n",
      "|    portfolio_value  | 1e+05    |\n",
      "|    total_cost       | 0        |\n",
      "|    total_reward     | 0        |\n",
      "|    total_reward_pct | 0        |\n",
      "|    total_trades     | 0        |\n",
      "| time/               |          |\n",
      "|    episodes         | 8        |\n",
      "|    fps              | 145      |\n",
      "|    time_elapsed     | 138      |\n",
      "|    total timesteps  | 20128    |\n",
      "| train/              |          |\n",
      "|    actor_loss       | -96.5    |\n",
      "|    critic_loss      | 757      |\n",
      "|    learning_rate    | 0.0001   |\n",
      "|    n_updates        | 17612    |\n",
      "----------------------------------\n",
      "day: 2515, episode: 30\n",
      "begin_total_asset: 100000.00\n",
      "end_total_asset: 100000.00\n",
      "total_reward: 0.00\n",
      "total_cost: 0.00\n",
      "total_trades: 0\n",
      "=================================\n",
      "----------------------------------\n",
      "| environment/        |          |\n",
      "|    portfolio_value  | 1e+05    |\n",
      "|    total_cost       | 0        |\n",
      "|    total_reward     | 0        |\n",
      "|    total_reward_pct | 0        |\n",
      "|    total_trades     | 0        |\n",
      "| time/               |          |\n",
      "|    episodes         | 12       |\n",
      "|    fps              | 142      |\n",
      "|    time_elapsed     | 212      |\n",
      "|    total timesteps  | 30192    |\n",
      "| train/              |          |\n",
      "|    actor_loss       | -55      |\n",
      "|    critic_loss      | 621      |\n",
      "|    learning_rate    | 0.0001   |\n",
      "|    n_updates        | 27676    |\n",
      "----------------------------------\n"
     ]
    }
   ],
   "source": [
    "trained_ddpg = agent.train_model(model=model_ddpg, \n",
    "                             tb_log_name='ddpg',\n",
    "                             total_timesteps=30000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1sve9WGvsC__"
   },
   "source": [
    "### Model 3: PPO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BOjuycpS-Qvn",
    "outputId": "13ab0019-419b-4e6e-949d-c2617ae296b3"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'n_steps': 2048, 'ent_coef': 0.005, 'learning_rate': 0.0001, 'batch_size': 128}\nUsing cpu device\n"
     ]
    }
   ],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "PPO_PARAMS = {\n",
    "    \"n_steps\": 2048,\n",
    "    \"ent_coef\": 0.005,\n",
    "    \"learning_rate\": 0.0001,\n",
    "    \"batch_size\": 128,\n",
    "}\n",
    "model_ppo = agent.get_model(\"ppo\",model_kwargs = PPO_PARAMS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "G9hCHA8A-RSy",
    "outputId": "df865883-f3a8-42b2-951f-b129b44bd248"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Logging to tensorboard_log/ppo\\ppo_1\n",
      "-----------------------------\n",
      "| time/              |      |\n",
      "|    fps             | 1122 |\n",
      "|    iterations      | 1    |\n",
      "|    time_elapsed    | 1    |\n",
      "|    total_timesteps | 2048 |\n",
      "-----------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 1.09e+05     |\n",
      "|    total_cost           | 3.35e+03     |\n",
      "|    total_reward         | 8.7e+03      |\n",
      "|    total_reward_pct     | 8.7          |\n",
      "|    total_trades         | 2464         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 985          |\n",
      "|    iterations           | 2            |\n",
      "|    time_elapsed         | 4            |\n",
      "|    total_timesteps      | 4096         |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0011447369 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.42        |\n",
      "|    explained_variance   | nan          |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | -0.00955     |\n",
      "|    n_updates            | 10           |\n",
      "|    policy_gradient_loss | -3.34e-05    |\n",
      "|    std                  | 0.999        |\n",
      "|    value_loss           | 0.00355      |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 1.53e+05     |\n",
      "|    total_cost           | 3.3e+03      |\n",
      "|    total_reward         | 5.31e+04     |\n",
      "|    total_reward_pct     | 53.1         |\n",
      "|    total_trades         | 2479         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 947          |\n",
      "|    iterations           | 3            |\n",
      "|    time_elapsed         | 6            |\n",
      "|    total_timesteps      | 6144         |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0033182062 |\n",
      "|    clip_fraction        | 0.0141       |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.42        |\n",
      "|    explained_variance   | -9.56        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 0.0176       |\n",
      "|    n_updates            | 20           |\n",
      "|    policy_gradient_loss | -0.00147     |\n",
      "|    std                  | 1            |\n",
      "|    value_loss           | 0.0588       |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 1.83e+05     |\n",
      "|    total_cost           | 3.15e+03     |\n",
      "|    total_reward         | 8.3e+04      |\n",
      "|    total_reward_pct     | 83           |\n",
      "|    total_trades         | 2444         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 927          |\n",
      "|    iterations           | 4            |\n",
      "|    time_elapsed         | 8            |\n",
      "|    total_timesteps      | 8192         |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0015491298 |\n",
      "|    clip_fraction        | 0.00791      |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.42        |\n",
      "|    explained_variance   | -88.7        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 0.00955      |\n",
      "|    n_updates            | 30           |\n",
      "|    policy_gradient_loss | -0.000997    |\n",
      "|    std                  | 1            |\n",
      "|    value_loss           | 0.0601       |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 1.25e+05     |\n",
      "|    total_cost           | 3.12e+03     |\n",
      "|    total_reward         | 2.52e+04     |\n",
      "|    total_reward_pct     | 25.2         |\n",
      "|    total_trades         | 2426         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 917          |\n",
      "|    iterations           | 5            |\n",
      "|    time_elapsed         | 11           |\n",
      "|    total_timesteps      | 10240        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0025092566 |\n",
      "|    clip_fraction        | 0.00449      |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.42        |\n",
      "|    explained_variance   | -1.12        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 0.123        |\n",
      "|    n_updates            | 40           |\n",
      "|    policy_gradient_loss | -0.00147     |\n",
      "|    std                  | 1            |\n",
      "|    value_loss           | 0.256        |\n",
      "------------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 913         |\n",
      "|    iterations           | 6           |\n",
      "|    time_elapsed         | 13          |\n",
      "|    total_timesteps      | 12288       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.007703814 |\n",
      "|    clip_fraction        | 0.0283      |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.42       |\n",
      "|    explained_variance   | -2.28e+13   |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 0.00854     |\n",
      "|    n_updates            | 50          |\n",
      "|    policy_gradient_loss | -0.00168    |\n",
      "|    std                  | 1           |\n",
      "|    value_loss           | 0.00509     |\n",
      "-----------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 1.98e+05     |\n",
      "|    total_cost           | 3.33e+03     |\n",
      "|    total_reward         | 9.81e+04     |\n",
      "|    total_reward_pct     | 98.1         |\n",
      "|    total_trades         | 2479         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 908          |\n",
      "|    iterations           | 7            |\n",
      "|    time_elapsed         | 15           |\n",
      "|    total_timesteps      | 14336        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0063809976 |\n",
      "|    clip_fraction        | 0.0454       |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.42        |\n",
      "|    explained_variance   | -0.954       |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 0.0629       |\n",
      "|    n_updates            | 60           |\n",
      "|    policy_gradient_loss | -0.00356     |\n",
      "|    std                  | 1            |\n",
      "|    value_loss           | 0.13         |\n",
      "------------------------------------------\n",
      "day: 2515, episode: 40\n",
      "begin_total_asset: 100000.00\n",
      "end_total_asset: 102184.69\n",
      "total_reward: 2184.69\n",
      "total_cost: 3061.24\n",
      "total_trades: 2347\n",
      "Sharpe: 0.080\n",
      "=================================\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 1.02e+05    |\n",
      "|    total_cost           | 3.06e+03    |\n",
      "|    total_reward         | 2.18e+03    |\n",
      "|    total_reward_pct     | 2.18        |\n",
      "|    total_trades         | 2347        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 904         |\n",
      "|    iterations           | 8           |\n",
      "|    time_elapsed         | 18          |\n",
      "|    total_timesteps      | 16384       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.004123468 |\n",
      "|    clip_fraction        | 0.00483     |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.42       |\n",
      "|    explained_variance   | nan         |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 0.0319      |\n",
      "|    n_updates            | 70          |\n",
      "|    policy_gradient_loss | -0.00059    |\n",
      "|    std                  | 1           |\n",
      "|    value_loss           | 0.0979      |\n",
      "-----------------------------------------\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 1.06e+05    |\n",
      "|    total_cost           | 2.63e+03    |\n",
      "|    total_reward         | 6.11e+03    |\n",
      "|    total_reward_pct     | 6.11        |\n",
      "|    total_trades         | 2190        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 901         |\n",
      "|    iterations           | 9           |\n",
      "|    time_elapsed         | 20          |\n",
      "|    total_timesteps      | 18432       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.011966931 |\n",
      "|    clip_fraction        | 0.0411      |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.42       |\n",
      "|    explained_variance   | -1.76e+13   |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | -0.00819    |\n",
      "|    n_updates            | 80          |\n",
      "|    policy_gradient_loss | -0.00214    |\n",
      "|    std                  | 0.997       |\n",
      "|    value_loss           | 0.00394     |\n",
      "-----------------------------------------\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 1.06e+05    |\n",
      "|    total_cost           | 2.85e+03    |\n",
      "|    total_reward         | 6.2e+03     |\n",
      "|    total_reward_pct     | 6.2         |\n",
      "|    total_trades         | 2202        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 898         |\n",
      "|    iterations           | 10          |\n",
      "|    time_elapsed         | 22          |\n",
      "|    total_timesteps      | 20480       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.008785707 |\n",
      "|    clip_fraction        | 0.097       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.41       |\n",
      "|    explained_variance   | nan         |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 0.00322     |\n",
      "|    n_updates            | 90          |\n",
      "|    policy_gradient_loss | -0.00658    |\n",
      "|    std                  | 0.987       |\n",
      "|    value_loss           | 0.000598    |\n",
      "-----------------------------------------\n",
      "--------------------------------------------\n",
      "| time/                   |                |\n",
      "|    fps                  | 897            |\n",
      "|    iterations           | 11             |\n",
      "|    time_elapsed         | 25             |\n",
      "|    total_timesteps      | 22528          |\n",
      "| train/                  |                |\n",
      "|    approx_kl            | -2.6509253e-05 |\n",
      "|    clip_fraction        | 0              |\n",
      "|    clip_range           | 0.2            |\n",
      "|    entropy_loss         | -1.4           |\n",
      "|    explained_variance   | -1.25e+13      |\n",
      "|    learning_rate        | 0.0001         |\n",
      "|    loss                 | -0.00579       |\n",
      "|    n_updates            | 100            |\n",
      "|    policy_gradient_loss | 4.35e-05       |\n",
      "|    std                  | 0.983          |\n",
      "|    value_loss           | 0.0028         |\n",
      "--------------------------------------------\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 1.12e+05    |\n",
      "|    total_cost           | 3.01e+03    |\n",
      "|    total_reward         | 1.16e+04    |\n",
      "|    total_reward_pct     | 11.6        |\n",
      "|    total_trades         | 2324        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 895         |\n",
      "|    iterations           | 12          |\n",
      "|    time_elapsed         | 27          |\n",
      "|    total_timesteps      | 24576       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.005432145 |\n",
      "|    clip_fraction        | 0.065       |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.4        |\n",
      "|    explained_variance   | nan         |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | -0.00813    |\n",
      "|    n_updates            | 110         |\n",
      "|    policy_gradient_loss | -0.0055     |\n",
      "|    std                  | 0.98        |\n",
      "|    value_loss           | 0.00267     |\n",
      "-----------------------------------------\n",
      "-------------------------------------------\n",
      "| environment/            |               |\n",
      "|    portfolio_value      | 3.08e+05      |\n",
      "|    total_cost           | 3.13e+03      |\n",
      "|    total_reward         | 2.08e+05      |\n",
      "|    total_reward_pct     | 208           |\n",
      "|    total_trades         | 2485          |\n",
      "| time/                   |               |\n",
      "|    fps                  | 894           |\n",
      "|    iterations           | 13            |\n",
      "|    time_elapsed         | 29            |\n",
      "|    total_timesteps      | 26624         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00022748549 |\n",
      "|    clip_fraction        | 0.00156       |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.4          |\n",
      "|    explained_variance   | -3.88         |\n",
      "|    learning_rate        | 0.0001        |\n",
      "|    loss                 | 0.138         |\n",
      "|    n_updates            | 120           |\n",
      "|    policy_gradient_loss | -0.000759     |\n",
      "|    std                  | 0.978         |\n",
      "|    value_loss           | 0.274         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 3.62e+05     |\n",
      "|    total_cost           | 2.73e+03     |\n",
      "|    total_reward         | 2.62e+05     |\n",
      "|    total_reward_pct     | 262          |\n",
      "|    total_trades         | 2496         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 893          |\n",
      "|    iterations           | 14           |\n",
      "|    time_elapsed         | 32           |\n",
      "|    total_timesteps      | 28672        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0040495247 |\n",
      "|    clip_fraction        | 0.00967      |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.4         |\n",
      "|    explained_variance   | -15.8        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 0.318        |\n",
      "|    n_updates            | 130          |\n",
      "|    policy_gradient_loss | -0.00205     |\n",
      "|    std                  | 0.98         |\n",
      "|    value_loss           | 0.794        |\n",
      "------------------------------------------\n",
      "--------------------------------------------\n",
      "| environment/            |                |\n",
      "|    portfolio_value      | 3.33e+05       |\n",
      "|    total_cost           | 2.62e+03       |\n",
      "|    total_reward         | 2.33e+05       |\n",
      "|    total_reward_pct     | 233            |\n",
      "|    total_trades         | 2476           |\n",
      "| time/                   |                |\n",
      "|    fps                  | 892            |\n",
      "|    iterations           | 15             |\n",
      "|    time_elapsed         | 34             |\n",
      "|    total_timesteps      | 30720          |\n",
      "| train/                  |                |\n",
      "|    approx_kl            | -0.00018516646 |\n",
      "|    clip_fraction        | 0              |\n",
      "|    clip_range           | 0.2            |\n",
      "|    entropy_loss         | -1.4           |\n",
      "|    explained_variance   | -15.1          |\n",
      "|    learning_rate        | 0.0001         |\n",
      "|    loss                 | 0.495          |\n",
      "|    n_updates            | 140            |\n",
      "|    policy_gradient_loss | -0.000216      |\n",
      "|    std                  | 0.979          |\n",
      "|    value_loss           | 1.37           |\n",
      "--------------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 3.85e+05     |\n",
      "|    total_cost           | 2.45e+03     |\n",
      "|    total_reward         | 2.85e+05     |\n",
      "|    total_reward_pct     | 285          |\n",
      "|    total_trades         | 2487         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 891          |\n",
      "|    iterations           | 16           |\n",
      "|    time_elapsed         | 36           |\n",
      "|    total_timesteps      | 32768        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0032079243 |\n",
      "|    clip_fraction        | 0.00542      |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.4         |\n",
      "|    explained_variance   | -21.9        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 0.34         |\n",
      "|    n_updates            | 150          |\n",
      "|    policy_gradient_loss | -0.00282     |\n",
      "|    std                  | 0.976        |\n",
      "|    value_loss           | 1.03         |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 890           |\n",
      "|    iterations           | 17            |\n",
      "|    time_elapsed         | 39            |\n",
      "|    total_timesteps      | 34816         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | -8.615013e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.39         |\n",
      "|    explained_variance   | -12           |\n",
      "|    learning_rate        | 0.0001        |\n",
      "|    loss                 | 0.993         |\n",
      "|    n_updates            | 160           |\n",
      "|    policy_gradient_loss | -0.000598     |\n",
      "|    std                  | 0.976         |\n",
      "|    value_loss           | 1.76          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 4.86e+05     |\n",
      "|    total_cost           | 2.17e+03     |\n",
      "|    total_reward         | 3.86e+05     |\n",
      "|    total_reward_pct     | 386          |\n",
      "|    total_trades         | 2496         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 889          |\n",
      "|    iterations           | 18           |\n",
      "|    time_elapsed         | 41           |\n",
      "|    total_timesteps      | 36864        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0012377047 |\n",
      "|    clip_fraction        | 0.000244     |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.39        |\n",
      "|    explained_variance   | -7.21        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 0.589        |\n",
      "|    n_updates            | 170          |\n",
      "|    policy_gradient_loss | -0.000287    |\n",
      "|    std                  | 0.975        |\n",
      "|    value_loss           | 1.16         |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| environment/            |               |\n",
      "|    portfolio_value      | 4.87e+05      |\n",
      "|    total_cost           | 1.85e+03      |\n",
      "|    total_reward         | 3.87e+05      |\n",
      "|    total_reward_pct     | 387           |\n",
      "|    total_trades         | 2497          |\n",
      "| time/                   |               |\n",
      "|    fps                  | 888           |\n",
      "|    iterations           | 19            |\n",
      "|    time_elapsed         | 43            |\n",
      "|    total_timesteps      | 38912         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00045283558 |\n",
      "|    clip_fraction        | 0.0116        |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.39         |\n",
      "|    explained_variance   | -11.5         |\n",
      "|    learning_rate        | 0.0001        |\n",
      "|    loss                 | 1.04          |\n",
      "|    n_updates            | 180           |\n",
      "|    policy_gradient_loss | -0.00176      |\n",
      "|    std                  | 0.974         |\n",
      "|    value_loss           | 2.29          |\n",
      "-------------------------------------------\n",
      "day: 2515, episode: 50\n",
      "begin_total_asset: 100000.00\n",
      "end_total_asset: 474949.78\n",
      "total_reward: 374949.78\n",
      "total_cost: 1779.09\n",
      "total_trades: 2480\n",
      "Sharpe: 0.800\n",
      "=================================\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 4.75e+05     |\n",
      "|    total_cost           | 1.78e+03     |\n",
      "|    total_reward         | 3.75e+05     |\n",
      "|    total_reward_pct     | 375          |\n",
      "|    total_trades         | 2480         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 888          |\n",
      "|    iterations           | 20           |\n",
      "|    time_elapsed         | 46           |\n",
      "|    total_timesteps      | 40960        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0014295809 |\n",
      "|    clip_fraction        | 0.00112      |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.39        |\n",
      "|    explained_variance   | -9.7         |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 0.871        |\n",
      "|    n_updates            | 190          |\n",
      "|    policy_gradient_loss | -0.00027     |\n",
      "|    std                  | 0.974        |\n",
      "|    value_loss           | 2.58         |\n",
      "------------------------------------------\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 4.64e+05    |\n",
      "|    total_cost           | 1.64e+03    |\n",
      "|    total_reward         | 3.64e+05    |\n",
      "|    total_reward_pct     | 364         |\n",
      "|    total_trades         | 2506        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 887         |\n",
      "|    iterations           | 21          |\n",
      "|    time_elapsed         | 48          |\n",
      "|    total_timesteps      | 43008       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.007133349 |\n",
      "|    clip_fraction        | 0.0312      |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.39       |\n",
      "|    explained_variance   | -12.3       |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 1.17        |\n",
      "|    n_updates            | 200         |\n",
      "|    policy_gradient_loss | -0.0032     |\n",
      "|    std                  | 0.975       |\n",
      "|    value_loss           | 2.4         |\n",
      "-----------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 887          |\n",
      "|    iterations           | 22           |\n",
      "|    time_elapsed         | 50           |\n",
      "|    total_timesteps      | 45056        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0036850034 |\n",
      "|    clip_fraction        | 0.00215      |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.39        |\n",
      "|    explained_variance   | -25.1        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 1.36         |\n",
      "|    n_updates            | 210          |\n",
      "|    policy_gradient_loss | -0.000548    |\n",
      "|    std                  | 0.977        |\n",
      "|    value_loss           | 2.59         |\n",
      "------------------------------------------\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 5.62e+05    |\n",
      "|    total_cost           | 1.38e+03    |\n",
      "|    total_reward         | 4.62e+05    |\n",
      "|    total_reward_pct     | 462         |\n",
      "|    total_trades         | 2504        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 886         |\n",
      "|    iterations           | 23          |\n",
      "|    time_elapsed         | 53          |\n",
      "|    total_timesteps      | 47104       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.002375051 |\n",
      "|    clip_fraction        | 0.0179      |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.4        |\n",
      "|    explained_variance   | -21.1       |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 0.86        |\n",
      "|    n_updates            | 220         |\n",
      "|    policy_gradient_loss | -0.00118    |\n",
      "|    std                  | 0.975       |\n",
      "|    value_loss           | 1.83        |\n",
      "-----------------------------------------\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 5.65e+05    |\n",
      "|    total_cost           | 1.28e+03    |\n",
      "|    total_reward         | 4.65e+05    |\n",
      "|    total_reward_pct     | 465         |\n",
      "|    total_trades         | 2498        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 885         |\n",
      "|    iterations           | 24          |\n",
      "|    time_elapsed         | 55          |\n",
      "|    total_timesteps      | 49152       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.004144217 |\n",
      "|    clip_fraction        | 0.00171     |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.39       |\n",
      "|    explained_variance   | -17.5       |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 2.07        |\n",
      "|    n_updates            | 230         |\n",
      "|    policy_gradient_loss | -0.000279   |\n",
      "|    std                  | 0.976       |\n",
      "|    value_loss           | 3.18        |\n",
      "-----------------------------------------\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 6.55e+05    |\n",
      "|    total_cost           | 1.08e+03    |\n",
      "|    total_reward         | 5.55e+05    |\n",
      "|    total_reward_pct     | 555         |\n",
      "|    total_trades         | 2497        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 885         |\n",
      "|    iterations           | 25          |\n",
      "|    time_elapsed         | 57          |\n",
      "|    total_timesteps      | 51200       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.004585681 |\n",
      "|    clip_fraction        | 0.0124      |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.39       |\n",
      "|    explained_variance   | -28.3       |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 1.76        |\n",
      "|    n_updates            | 240         |\n",
      "|    policy_gradient_loss | -0.0014     |\n",
      "|    std                  | 0.974       |\n",
      "|    value_loss           | 3.52        |\n",
      "-----------------------------------------\n",
      "-------------------------------------------\n",
      "| environment/            |               |\n",
      "|    portfolio_value      | 6.34e+05      |\n",
      "|    total_cost           | 785           |\n",
      "|    total_reward         | 5.34e+05      |\n",
      "|    total_reward_pct     | 534           |\n",
      "|    total_trades         | 2502          |\n",
      "| time/                   |               |\n",
      "|    fps                  | 885           |\n",
      "|    iterations           | 26            |\n",
      "|    time_elapsed         | 60            |\n",
      "|    total_timesteps      | 53248         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | -0.0012456017 |\n",
      "|    clip_fraction        | 0.00269       |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.39         |\n",
      "|    explained_variance   | -29.8         |\n",
      "|    learning_rate        | 0.0001        |\n",
      "|    loss                 | 2.33          |\n",
      "|    n_updates            | 250           |\n",
      "|    policy_gradient_loss | -0.000415     |\n",
      "|    std                  | 0.979         |\n",
      "|    value_loss           | 4.68          |\n",
      "-------------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 885         |\n",
      "|    iterations           | 27          |\n",
      "|    time_elapsed         | 62          |\n",
      "|    total_timesteps      | 55296       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.006417418 |\n",
      "|    clip_fraction        | 0.00337     |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.4        |\n",
      "|    explained_variance   | -19.8       |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 2.31        |\n",
      "|    n_updates            | 260         |\n",
      "|    policy_gradient_loss | -0.000269   |\n",
      "|    std                  | 0.98        |\n",
      "|    value_loss           | 4.67        |\n",
      "-----------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 5.79e+05     |\n",
      "|    total_cost           | 1.07e+03     |\n",
      "|    total_reward         | 4.79e+05     |\n",
      "|    total_reward_pct     | 479          |\n",
      "|    total_trades         | 2500         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 884          |\n",
      "|    iterations           | 28           |\n",
      "|    time_elapsed         | 64           |\n",
      "|    total_timesteps      | 57344        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0024222317 |\n",
      "|    clip_fraction        | 0.000879     |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.4         |\n",
      "|    explained_variance   | -227         |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 1.29         |\n",
      "|    n_updates            | 270          |\n",
      "|    policy_gradient_loss | -5.68e-05    |\n",
      "|    std                  | 0.981        |\n",
      "|    value_loss           | 2.66         |\n",
      "------------------------------------------\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 6.28e+05    |\n",
      "|    total_cost           | 932         |\n",
      "|    total_reward         | 5.28e+05    |\n",
      "|    total_reward_pct     | 528         |\n",
      "|    total_trades         | 2502        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 884         |\n",
      "|    iterations           | 29          |\n",
      "|    time_elapsed         | 67          |\n",
      "|    total_timesteps      | 59392       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.006682484 |\n",
      "|    clip_fraction        | 0.0406      |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.4        |\n",
      "|    explained_variance   | -17         |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 1.51        |\n",
      "|    n_updates            | 280         |\n",
      "|    policy_gradient_loss | -0.00435    |\n",
      "|    std                  | 0.979       |\n",
      "|    value_loss           | 2.96        |\n",
      "-----------------------------------------\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 6.44e+05    |\n",
      "|    total_cost           | 735         |\n",
      "|    total_reward         | 5.44e+05    |\n",
      "|    total_reward_pct     | 544         |\n",
      "|    total_trades         | 2500        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 884         |\n",
      "|    iterations           | 30          |\n",
      "|    time_elapsed         | 69          |\n",
      "|    total_timesteps      | 61440       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.001981568 |\n",
      "|    clip_fraction        | 0.0158      |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.4        |\n",
      "|    explained_variance   | -29.5       |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 1.63        |\n",
      "|    n_updates            | 290         |\n",
      "|    policy_gradient_loss | -0.00104    |\n",
      "|    std                  | 0.977       |\n",
      "|    value_loss           | 4.05        |\n",
      "-----------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 6.57e+05     |\n",
      "|    total_cost           | 606          |\n",
      "|    total_reward         | 5.57e+05     |\n",
      "|    total_reward_pct     | 557          |\n",
      "|    total_trades         | 2499         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 884          |\n",
      "|    iterations           | 31           |\n",
      "|    time_elapsed         | 71           |\n",
      "|    total_timesteps      | 63488        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0005870234 |\n",
      "|    clip_fraction        | 0.000977     |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.39        |\n",
      "|    explained_variance   | -28.2        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 1.97         |\n",
      "|    n_updates            | 300          |\n",
      "|    policy_gradient_loss | -5.17e-05    |\n",
      "|    std                  | 0.976        |\n",
      "|    value_loss           | 4.74         |\n",
      "------------------------------------------\n",
      "day: 2515, episode: 60\n",
      "begin_total_asset: 100000.00\n",
      "end_total_asset: 694355.78\n",
      "total_reward: 594355.78\n",
      "total_cost: 606.67\n",
      "total_trades: 2509\n",
      "Sharpe: 0.919\n",
      "=================================\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 6.94e+05     |\n",
      "|    total_cost           | 607          |\n",
      "|    total_reward         | 5.94e+05     |\n",
      "|    total_reward_pct     | 594          |\n",
      "|    total_trades         | 2509         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 883          |\n",
      "|    iterations           | 32           |\n",
      "|    time_elapsed         | 74           |\n",
      "|    total_timesteps      | 65536        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0003196007 |\n",
      "|    clip_fraction        | 0.00259      |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.39        |\n",
      "|    explained_variance   | -37.6        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 1.54         |\n",
      "|    n_updates            | 310          |\n",
      "|    policy_gradient_loss | 0.000279     |\n",
      "|    std                  | 0.975        |\n",
      "|    value_loss           | 4.69         |\n",
      "------------------------------------------\n",
      "--------------------------------------------\n",
      "| time/                   |                |\n",
      "|    fps                  | 883            |\n",
      "|    iterations           | 33             |\n",
      "|    time_elapsed         | 76             |\n",
      "|    total_timesteps      | 67584          |\n",
      "| train/                  |                |\n",
      "|    approx_kl            | -2.9055023e-05 |\n",
      "|    clip_fraction        | 0              |\n",
      "|    clip_range           | 0.2            |\n",
      "|    entropy_loss         | -1.39          |\n",
      "|    explained_variance   | -61.2          |\n",
      "|    learning_rate        | 0.0001         |\n",
      "|    loss                 | 3.27           |\n",
      "|    n_updates            | 320            |\n",
      "|    policy_gradient_loss | -0.000184      |\n",
      "|    std                  | 0.97           |\n",
      "|    value_loss           | 5.87           |\n",
      "--------------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 6.41e+05     |\n",
      "|    total_cost           | 624          |\n",
      "|    total_reward         | 5.41e+05     |\n",
      "|    total_reward_pct     | 541          |\n",
      "|    total_trades         | 2505         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 883          |\n",
      "|    iterations           | 34           |\n",
      "|    time_elapsed         | 78           |\n",
      "|    total_timesteps      | 69632        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0032248504 |\n",
      "|    clip_fraction        | 0.00186      |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.39        |\n",
      "|    explained_variance   | -18.3        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 1.17         |\n",
      "|    n_updates            | 330          |\n",
      "|    policy_gradient_loss | -0.000258    |\n",
      "|    std                  | 0.966        |\n",
      "|    value_loss           | 2.15         |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 6.8e+05      |\n",
      "|    total_cost           | 526          |\n",
      "|    total_reward         | 5.8e+05      |\n",
      "|    total_reward_pct     | 580          |\n",
      "|    total_trades         | 2505         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 883          |\n",
      "|    iterations           | 35           |\n",
      "|    time_elapsed         | 81           |\n",
      "|    total_timesteps      | 71680        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0015430602 |\n",
      "|    clip_fraction        | 0.000586     |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.39        |\n",
      "|    explained_variance   | -22.2        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 1.98         |\n",
      "|    n_updates            | 340          |\n",
      "|    policy_gradient_loss | 2.23e-05     |\n",
      "|    std                  | 0.967        |\n",
      "|    value_loss           | 4.33         |\n",
      "------------------------------------------\n",
      "-----------------------------------------\n",
      "| environment/            |             |\n",
      "|    portfolio_value      | 6.96e+05    |\n",
      "|    total_cost           | 629         |\n",
      "|    total_reward         | 5.96e+05    |\n",
      "|    total_reward_pct     | 596         |\n",
      "|    total_trades         | 2502        |\n",
      "| time/                   |             |\n",
      "|    fps                  | 883         |\n",
      "|    iterations           | 36          |\n",
      "|    time_elapsed         | 83          |\n",
      "|    total_timesteps      | 73728       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.006174787 |\n",
      "|    clip_fraction        | 0.0022      |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.39       |\n",
      "|    explained_variance   | -29.5       |\n",
      "|    learning_rate        | 0.0001      |\n",
      "|    loss                 | 3.08        |\n",
      "|    n_updates            | 350         |\n",
      "|    policy_gradient_loss | 0.000117    |\n",
      "|    std                  | 0.967       |\n",
      "|    value_loss           | 5.1         |\n",
      "-----------------------------------------\n",
      "-------------------------------------------\n",
      "| environment/            |               |\n",
      "|    portfolio_value      | 6.8e+05       |\n",
      "|    total_cost           | 691           |\n",
      "|    total_reward         | 5.8e+05       |\n",
      "|    total_reward_pct     | 580           |\n",
      "|    total_trades         | 2502          |\n",
      "| time/                   |               |\n",
      "|    fps                  | 883           |\n",
      "|    iterations           | 37            |\n",
      "|    time_elapsed         | 85            |\n",
      "|    total_timesteps      | 75776         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | -0.0006708663 |\n",
      "|    clip_fraction        | 0.0105        |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.39         |\n",
      "|    explained_variance   | -35.9         |\n",
      "|    learning_rate        | 0.0001        |\n",
      "|    loss                 | 2.65          |\n",
      "|    n_updates            | 360           |\n",
      "|    policy_gradient_loss | -0.000899     |\n",
      "|    std                  | 0.968         |\n",
      "|    value_loss           | 5.22          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 882          |\n",
      "|    iterations           | 38           |\n",
      "|    time_elapsed         | 88           |\n",
      "|    total_timesteps      | 77824        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0076179495 |\n",
      "|    clip_fraction        | 0.0312       |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.39        |\n",
      "|    explained_variance   | -30.3        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 2.5          |\n",
      "|    n_updates            | 370          |\n",
      "|    policy_gradient_loss | -0.0025      |\n",
      "|    std                  | 0.971        |\n",
      "|    value_loss           | 5.57         |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 6.62e+05     |\n",
      "|    total_cost           | 612          |\n",
      "|    total_reward         | 5.62e+05     |\n",
      "|    total_reward_pct     | 562          |\n",
      "|    total_trades         | 2507         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 882          |\n",
      "|    iterations           | 39           |\n",
      "|    time_elapsed         | 90           |\n",
      "|    total_timesteps      | 79872        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0013839847 |\n",
      "|    clip_fraction        | 0.0236       |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.39        |\n",
      "|    explained_variance   | -613         |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 1.61         |\n",
      "|    n_updates            | 380          |\n",
      "|    policy_gradient_loss | -0.00187     |\n",
      "|    std                  | 0.97         |\n",
      "|    value_loss           | 2.79         |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| environment/            |              |\n",
      "|    portfolio_value      | 6.49e+05     |\n",
      "|    total_cost           | 700          |\n",
      "|    total_reward         | 5.49e+05     |\n",
      "|    total_reward_pct     | 549          |\n",
      "|    total_trades         | 2504         |\n",
      "| time/                   |              |\n",
      "|    fps                  | 882          |\n",
      "|    iterations           | 40           |\n",
      "|    time_elapsed         | 92           |\n",
      "|    total_timesteps      | 81920        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0016418535 |\n",
      "|    clip_fraction        | 0.000244     |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.39        |\n",
      "|    explained_variance   | -17.5        |\n",
      "|    learning_rate        | 0.0001       |\n",
      "|    loss                 | 1.41         |\n",
      "|    n_updates            | 390          |\n",
      "|    policy_gradient_loss | 8.15e-05     |\n",
      "|    std                  | 0.966        |\n",
      "|    value_loss           | 4.27         |\n",
      "------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "trained_ppo = agent.train_model(model=model_ppo, \n",
    "                             tb_log_name='ppo',\n",
    "                             total_timesteps=80000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CiBG2ZknsG73"
   },
   "source": [
    "### Model 4: TD3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UcRdoBc8-Xze",
    "outputId": "cc0a65bf-f79d-48ce-c173-6ae08747b0fa"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'batch_size': 128, 'buffer_size': 1000000, 'learning_rate': 0.0003}\nUsing cpu device\n"
     ]
    }
   ],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "TD3_PARAMS = {\"batch_size\": 128, \n",
    "              \"buffer_size\": 1000000, \n",
    "              \"learning_rate\": 0.0003}\n",
    "\n",
    "model_td3 = agent.get_model(\"td3\",model_kwargs = TD3_PARAMS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CB5cAAio-ZSs",
    "outputId": "741a1ed9-9bb0-4763-f237-5cd461ac100e"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Logging to tensorboard_log/td3\\td3_1\n",
      "day: 2515, episode: 70\n",
      "begin_total_asset: 100000.00\n",
      "end_total_asset: 100000.00\n",
      "total_reward: 0.00\n",
      "total_cost: 0.00\n",
      "total_trades: 0\n",
      "=================================\n",
      "----------------------------------\n",
      "| environment/        |          |\n",
      "|    portfolio_value  | 1e+05    |\n",
      "|    total_cost       | 0        |\n",
      "|    total_reward     | 0        |\n",
      "|    total_reward_pct | 0        |\n",
      "|    total_trades     | 0        |\n",
      "| time/               |          |\n",
      "|    episodes         | 4        |\n",
      "|    fps              | 156      |\n",
      "|    time_elapsed     | 64       |\n",
      "|    total timesteps  | 10064    |\n",
      "| train/              |          |\n",
      "|    actor_loss       | 319      |\n",
      "|    critic_loss      | 6.01e+03 |\n",
      "|    learning_rate    | 0.0003   |\n",
      "|    n_updates        | 7548     |\n",
      "----------------------------------\n",
      "----------------------------------\n",
      "| environment/        |          |\n",
      "|    portfolio_value  | 1e+05    |\n",
      "|    total_cost       | 0        |\n",
      "|    total_reward     | 0        |\n",
      "|    total_reward_pct | 0        |\n",
      "|    total_trades     | 0        |\n",
      "| time/               |          |\n",
      "|    episodes         | 8        |\n",
      "|    fps              | 135      |\n",
      "|    time_elapsed     | 148      |\n",
      "|    total timesteps  | 20128    |\n",
      "| train/              |          |\n",
      "|    actor_loss       | 314      |\n",
      "|    critic_loss      | 1.51e+03 |\n",
      "|    learning_rate    | 0.0003   |\n",
      "|    n_updates        | 17612    |\n",
      "----------------------------------\n",
      "----------------------------------\n",
      "| environment/        |          |\n",
      "|    portfolio_value  | 1e+05    |\n",
      "|    total_cost       | 0        |\n",
      "|    total_reward     | 0        |\n",
      "|    total_reward_pct | 0        |\n",
      "|    total_trades     | 0        |\n",
      "| time/               |          |\n",
      "|    episodes         | 12       |\n",
      "|    fps              | 128      |\n",
      "|    time_elapsed     | 234      |\n",
      "|    total timesteps  | 30192    |\n",
      "| train/              |          |\n",
      "|    actor_loss       | 268      |\n",
      "|    critic_loss      | 282      |\n",
      "|    learning_rate    | 0.0003   |\n",
      "|    n_updates        | 27676    |\n",
      "----------------------------------\n"
     ]
    }
   ],
   "source": [
    "trained_td3 = agent.train_model(model=model_td3, \n",
    "                             tb_log_name='td3',\n",
    "                             total_timesteps=30000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oDk2qrlTLZCp"
   },
   "source": [
    "### Model 4: SAC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "el8sK4fo-dl1",
    "outputId": "20c5793c-397b-4dc3-a702-857178e61813"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'batch_size': 128, 'buffer_size': 100000, 'learning_rate': 3e-05, 'learning_starts': 100, 'ent_coef': 'auto_0.1'}\nUsing cpu device\n"
     ]
    }
   ],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "SAC_PARAMS = {\n",
    "    \"batch_size\": 128,\n",
    "    \"buffer_size\": 100000,\n",
    "    \"learning_rate\": 0.00003,\n",
    "    \"learning_starts\": 100,\n",
    "    \"ent_coef\": \"auto_0.1\",\n",
    "}\n",
    "\n",
    "model_sac = agent.get_model(\"sac\",model_kwargs = SAC_PARAMS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gaF80PR0-d_d",
    "outputId": "b26b6435-ca69-4000-b2b3-22596677d8ff"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Logging to tensorboard_log/sac\\sac_1\n",
      "----------------------------------\n",
      "| environment/        |          |\n",
      "|    portfolio_value  | 1e+05    |\n",
      "|    total_cost       | 0        |\n",
      "|    total_reward     | 0        |\n",
      "|    total_reward_pct | 0        |\n",
      "|    total_trades     | 0        |\n",
      "| time/               |          |\n",
      "|    episodes         | 4        |\n",
      "|    fps              | 85       |\n",
      "|    time_elapsed     | 117      |\n",
      "|    total timesteps  | 10064    |\n",
      "| train/              |          |\n",
      "|    actor_loss       | 1.21e+03 |\n",
      "|    critic_loss      | 1        |\n",
      "|    ent_coef         | 0.135    |\n",
      "|    ent_coef_loss    | 18.9     |\n",
      "|    learning_rate    | 3e-05    |\n",
      "|    n_updates        | 9963     |\n",
      "----------------------------------\n",
      "----------------------------------\n",
      "| environment/        |          |\n",
      "|    portfolio_value  | 1e+05    |\n",
      "|    total_cost       | 0        |\n",
      "|    total_reward     | 0        |\n",
      "|    total_reward_pct | 0        |\n",
      "|    total_trades     | 0        |\n",
      "| time/               |          |\n",
      "|    episodes         | 8        |\n",
      "|    fps              | 85       |\n",
      "|    time_elapsed     | 236      |\n",
      "|    total timesteps  | 20128    |\n",
      "| train/              |          |\n",
      "|    actor_loss       | 795      |\n",
      "|    critic_loss      | 1.84     |\n",
      "|    ent_coef         | 0.182    |\n",
      "|    ent_coef_loss    | 16       |\n",
      "|    learning_rate    | 3e-05    |\n",
      "|    n_updates        | 20027    |\n",
      "----------------------------------\n",
      "day: 2515, episode: 90\n",
      "begin_total_asset: 100000.00\n",
      "end_total_asset: 100000.00\n",
      "total_reward: 0.00\n",
      "total_cost: 0.00\n",
      "total_trades: 0\n",
      "=================================\n"
     ]
    }
   ],
   "source": [
    "trained_sac = agent.train_model(model=model_sac, \n",
    "                             tb_log_name='sac',\n",
    "                             total_timesteps=30000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KE1Xm9N9TnTn"
   },
   "source": [
    "### Trading\n",
    "* we use the environment class we initialized at 5.3 to create a stock trading environment\n",
    "* Assume that we have $100,000 initial capital at 2019-01-01. \n",
    "* We use the trained model of PPO to trade AAPL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 298
    },
    "id": "kLFUwxhCoHN_",
    "outputId": "22540921-fc5e-460b-f9ef-1438131b45d7"
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "         date       open       high        low      close     volume   tic  \\\n",
       "0  2019-01-02  38.722500  39.712502  38.557499  38.562561  148158800  AAPL   \n",
       "1  2019-01-03  35.994999  36.430000  35.500000  34.721451  365248800  AAPL   \n",
       "2  2019-01-04  36.132500  37.137501  35.950001  36.203678  234428400  AAPL   \n",
       "3  2019-01-07  37.174999  37.207500  36.474998  36.123104  219111200  AAPL   \n",
       "4  2019-01-08  37.389999  37.955002  37.130001  36.811718  164101200  AAPL   \n",
       "\n",
       "   day      macd    boll_ub  ...      dx_30  close_30_sma  close_60_sma  \\\n",
       "0    2 -2.019903  44.572026  ...  42.250808     41.287324     46.557656   \n",
       "1    3 -2.203028  43.977597  ...  55.246973     40.869433     46.226696   \n",
       "2    4 -2.203157  43.519698  ...  47.060632     40.563381     45.922549   \n",
       "3    0 -2.184578  43.067268  ...  46.245025     40.326923     45.604486   \n",
       "4    1 -2.090194  42.797282  ...  37.537680     40.115047     45.340525   \n",
       "\n",
       "        kdjk  open_2_sma       boll  close_10.0_le_5_c       wr_10       dma  \\\n",
       "0  27.327775   39.177500  40.034790                0.0   63.418114 -6.886014   \n",
       "1  13.056580   37.358749  39.514298                0.0  112.236531 -7.096225   \n",
       "2  14.108980   36.063749  39.167181                0.0   86.003419 -7.054847   \n",
       "3  14.191732   36.653749  38.840088                0.0   85.642763 -6.910649   \n",
       "4  19.535793   37.282499  38.623490                0.0   69.776084 -6.599589   \n",
       "\n",
       "       trix  \n",
       "0 -0.761652  \n",
       "1 -0.763466  \n",
       "2 -0.766086  \n",
       "3 -0.767321  \n",
       "4 -0.759067  \n",
       "\n",
       "[5 rows x 23 columns]"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>date</th>\n      <th>open</th>\n      <th>high</th>\n      <th>low</th>\n      <th>close</th>\n      <th>volume</th>\n      <th>tic</th>\n      <th>day</th>\n      <th>macd</th>\n      <th>boll_ub</th>\n      <th>...</th>\n      <th>dx_30</th>\n      <th>close_30_sma</th>\n      <th>close_60_sma</th>\n      <th>kdjk</th>\n      <th>open_2_sma</th>\n      <th>boll</th>\n      <th>close_10.0_le_5_c</th>\n      <th>wr_10</th>\n      <th>dma</th>\n      <th>trix</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2019-01-02</td>\n      <td>38.722500</td>\n      <td>39.712502</td>\n      <td>38.557499</td>\n      <td>38.562561</td>\n      <td>148158800</td>\n      <td>AAPL</td>\n      <td>2</td>\n      <td>-2.019903</td>\n      <td>44.572026</td>\n      <td>...</td>\n      <td>42.250808</td>\n      <td>41.287324</td>\n      <td>46.557656</td>\n      <td>27.327775</td>\n      <td>39.177500</td>\n      <td>40.034790</td>\n      <td>0.0</td>\n      <td>63.418114</td>\n      <td>-6.886014</td>\n      <td>-0.761652</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2019-01-03</td>\n      <td>35.994999</td>\n      <td>36.430000</td>\n      <td>35.500000</td>\n      <td>34.721451</td>\n      <td>365248800</td>\n      <td>AAPL</td>\n      <td>3</td>\n      <td>-2.203028</td>\n      <td>43.977597</td>\n      <td>...</td>\n      <td>55.246973</td>\n      <td>40.869433</td>\n      <td>46.226696</td>\n      <td>13.056580</td>\n      <td>37.358749</td>\n      <td>39.514298</td>\n      <td>0.0</td>\n      <td>112.236531</td>\n      <td>-7.096225</td>\n      <td>-0.763466</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2019-01-04</td>\n      <td>36.132500</td>\n      <td>37.137501</td>\n      <td>35.950001</td>\n      <td>36.203678</td>\n      <td>234428400</td>\n      <td>AAPL</td>\n      <td>4</td>\n      <td>-2.203157</td>\n      <td>43.519698</td>\n      <td>...</td>\n      <td>47.060632</td>\n      <td>40.563381</td>\n      <td>45.922549</td>\n      <td>14.108980</td>\n      <td>36.063749</td>\n      <td>39.167181</td>\n      <td>0.0</td>\n      <td>86.003419</td>\n      <td>-7.054847</td>\n      <td>-0.766086</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2019-01-07</td>\n      <td>37.174999</td>\n      <td>37.207500</td>\n      <td>36.474998</td>\n      <td>36.123104</td>\n      <td>219111200</td>\n      <td>AAPL</td>\n      <td>0</td>\n      <td>-2.184578</td>\n      <td>43.067268</td>\n      <td>...</td>\n      <td>46.245025</td>\n      <td>40.326923</td>\n      <td>45.604486</td>\n      <td>14.191732</td>\n      <td>36.653749</td>\n      <td>38.840088</td>\n      <td>0.0</td>\n      <td>85.642763</td>\n      <td>-6.910649</td>\n      <td>-0.767321</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2019-01-08</td>\n      <td>37.389999</td>\n      <td>37.955002</td>\n      <td>37.130001</td>\n      <td>36.811718</td>\n      <td>164101200</td>\n      <td>AAPL</td>\n      <td>1</td>\n      <td>-2.090194</td>\n      <td>42.797282</td>\n      <td>...</td>\n      <td>37.537680</td>\n      <td>40.115047</td>\n      <td>45.340525</td>\n      <td>19.535793</td>\n      <td>37.282499</td>\n      <td>38.623490</td>\n      <td>0.0</td>\n      <td>69.776084</td>\n      <td>-6.599589</td>\n      <td>-0.759067</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 23 columns</p>\n</div>"
     },
     "metadata": {},
     "execution_count": 31
    }
   ],
   "source": [
    "trade.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "bLHl6V7eqV6_"
   },
   "outputs": [
    {
     "output_type": "error",
     "ename": "TypeError",
     "evalue": "DRL_prediction() got an unexpected keyword argument 'trade_data'",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-38-52459e78ba62>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0menv_trade\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mobs_trade\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0me_trade_gym\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_sb_env\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 7\u001b[1;33m df_account_value, df_actions = DRLAgent.DRL_prediction(model=trained_sac,\n\u001b[0m\u001b[0;32m      8\u001b[0m                                            \u001b[0mtrade_data\u001b[0m  \u001b[1;33m=\u001b[0m \u001b[0mtrade\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m                                            \u001b[0mtest_env\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0menv_trade\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: DRL_prediction() got an unexpected keyword argument 'trade_data'"
     ]
    }
   ],
   "source": [
    "## make a prediction and get the account value change\n",
    "trade = data_split(data_df, start = '2019-01-01', end = '2021-01-01')\n",
    "\n",
    "e_trade_gym = StockTradingEnv(df = trade, **env_kwargs)\n",
    "env_trade, obs_trade = e_trade_gym.get_sb_env()\n",
    "\n",
    "df_account_value, df_actions = DRLAgent.DRL_prediction(model=trained_sac,\n",
    "                                           trade_data  = trade,\n",
    "                                           test_env = env_trade,\n",
    "                                           test_obs = obs_trade)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qYXxFzD5TnTw"
   },
   "source": [
    "<a id='6'></a>\n",
    "# Part 7: Backtesting Performance\n",
    "Backtesting plays a key role in evaluating the performance of a trading strategy. Automated backtesting tool is preferred because it reduces the human error. We usually use the Quantopian pyfolio package to backtest our trading strategies. It is easy to use and consists of various individual plots that provide a comprehensive image of the performance of a trading strategy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8GwqOO-v1NVz"
   },
   "source": [
    "<a id='6.1'></a>\n",
    "## 7.1 BackTestStats\n",
    "pass in df_account_value, this information is stored in env class\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "C9cpPm9YYxHC",
    "outputId": "7c7a8e34-7ed3-441d-9036-829240430843"
   },
   "outputs": [],
   "source": [
    "print(\"==============Get Backtest Results===========\")\n",
    "now = datetime.datetime.now().strftime('%Y%m%d-%Hh%M')\n",
    "\n",
    "perf_stats_all = BackTestStats(account_value=df_account_value)\n",
    "perf_stats_all = pd.DataFrame(perf_stats_all)\n",
    "perf_stats_all.to_csv(\"./\"+config.RESULTS_DIR+\"/perf_stats_all_\"+now+'.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "e-Tenjb0hcNr"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y4Gw3HNr1TDU"
   },
   "source": [
    "<a id='6.2'></a>\n",
    "## 7.2 BackTestPlot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "MA_8LuZE1J3X",
    "outputId": "3d2c46b2-2b6e-4cfd-9648-68ae7d37d89c"
   },
   "outputs": [],
   "source": [
    "print(\"==============Compare to AAPL itself buy-and-hold===========\")\n",
    "%matplotlib inline\n",
    "BackTestPlot(account_value=df_account_value, \n",
    "             baseline_ticker = 'AAPL',\n",
    "             baseline_start = '2019-01-01',\n",
    "             baseline_end = '2021-01-01')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qSFMdgCJE4O-"
   },
   "source": [
    "<a id='6.3'></a>\n",
    "## 7.3 Baseline Stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MpR7aQIwqdC4",
    "outputId": "24029a71-3596-4090-d7bc-7f44dd6000fc"
   },
   "outputs": [],
   "source": [
    "print(\"==============Get Baseline Stats===========\")\n",
    "baesline_perf_stats=BaselineStats('AAPL')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vSFRXQfYFTQf",
    "outputId": "a5951bab-aa47-4fa8-9cd9-9fc62b5b02f5"
   },
   "outputs": [],
   "source": [
    "print(\"==============Get Baseline Stats===========\")\n",
    "baesline_perf_stats=BaselineStats('^GSPC')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tunrA4mjE9la"
   },
   "source": [
    "<a id='6.4'></a>\n",
    "## 7.4 Compare to Stock Market Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "5Ca2gHxi1gzX",
    "outputId": "68de7e8c-882a-4e8a-c9a9-cea8ea6ca212"
   },
   "outputs": [],
   "source": [
    "print(\"==============Compare to S&P 500===========\")\n",
    "%matplotlib inline\n",
    "# S&P 500: ^GSPC\n",
    "# Dow Jones Index: ^DJI\n",
    "# NASDAQ 100: ^NDX\n",
    "BackTestPlot(df_account_value, baseline_ticker = '^GSPC')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3_aHGEOTCluG"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "Y9CM5DeIr9GC",
    "9upN8FI2r_X1",
    "CiBG2ZknsG73"
   ],
   "include_colab_link": true,
   "name": "FinRL_single_stock_trading.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}